You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by John Williams <wi...@tni.com> on 2004/10/07 22:37:00 UTC
slow status
I am trying to put a large website project into subversion-1.1.0.
fsfs is a big improvement, but there are still scalability problems.
Here is an especially obvious one.
$ date; svn status newsite.html ; date
Thu Oct 7 16:28:23 MDT 2004
Thu Oct 7 16:30:34 MDT 2004
2 minutes(!) for an svn status command on a single file.
I am using svnserve, but that doesn't matter.
An strace on the process shows that svn status is reading *every*
.svn/format and .svn/entries file in the entire project, starting at .
(the current directory), even though I only asked for the status of a
single regular file.
~ John Williams
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: slow status
Posted by kf...@collab.net.
John Williams <wi...@tni.com> writes:
> On 9 Oct 2004 kfogel@collab.net wrote:
> > John Williams <wi...@tni.com> writes:
> > > An strace on the process shows that svn status is reading *every*
> > > .svn/format and .svn/entries file in the entire project, starting at
> > > . (the current directory), even though I only asked for the status of
> > > a single regular file.
> >
> > Is your client (as well as the server side) also 1.1.0? I would have
> > thought this was
> >
> > http://subversion.tigris.org/issues/show_bug.cgi?id=1680
> >
> > ...but that issue is marked as resolved in 1.1.0.
>
> Yes. Client and server were compiled from the same source.
Okay, thanks.
As this problem is now being discussed on the dev@ list, in
http://subversion.tigris.org/servlets/BrowseList?list=dev&by=thread&from=242963
...I won't post any more over here about it, to avoid confusion.
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: slow status
Posted by John Williams <wi...@tni.com>.
On 9 Oct 2004 kfogel@collab.net wrote:
> John Williams <wi...@tni.com> writes:
> > An strace on the process shows that svn status is reading *every*
> > .svn/format and .svn/entries file in the entire project, starting at
> > . (the current directory), even though I only asked for the status of
> > a single regular file.
>
> Is your client (as well as the server side) also 1.1.0? I would have
> thought this was
>
> http://subversion.tigris.org/issues/show_bug.cgi?id=1680
>
> ...but that issue is marked as resolved in 1.1.0.
Yes. Client and server were compiled from the same source.
$ svn --version
svn, version 1.1.0 (r11180)
compiled Oct 1 2004, 15:14:08
Copyright (C) 2000-2004 CollabNet.
Subversion is open source software, see http://subversion.tigris.org/
This product includes software developed by CollabNet
(http://www.Collab.Net/).
The following repository access (RA) modules are available:
* ra_dav : Module for accessing a repository via WebDAV (DeltaV) protocol.
- handles 'http' schema
* ra_local : Module for accessing a repository on local disk.
- handles 'file' schema
* ra_svn : Module for accessing a repository using the svn network
protocol.
- handles 'svn' schema
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: slow status
Posted by kf...@collab.net.
John Williams <wi...@tni.com> writes:
> I am trying to put a large website project into subversion-1.1.0.
> fsfs is a big improvement, but there are still scalability problems.
>
> Here is an especially obvious one.
>
> $ date; svn status newsite.html ; date
> Thu Oct 7 16:28:23 MDT 2004
> Thu Oct 7 16:30:34 MDT 2004
>
> 2 minutes(!) for an svn status command on a single file.
> I am using svnserve, but that doesn't matter.
>
> An strace on the process shows that svn status is reading *every*
> .svn/format and .svn/entries file in the entire project, starting at
> . (the current directory), even though I only asked for the status of
> a single regular file.
Is your client (as well as the server side) also 1.1.0? I would have
thought this was
http://subversion.tigris.org/issues/show_bug.cgi?id=1680
...but that issue is marked as resolved in 1.1.0.
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: slow status
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
On Thu, 7 Oct 2004, Greg Hudson wrote:
> On Thu, 2004-10-07 at 19:57, Ben Collins-Sussman wrote:
> > $ svn status subversion/libsvn_fs_base/bdb/dbt.c
> >
> > Then I only see the bdb/.svn/entries file being opened.
> >
> > But when I run
> >
> > $ svn status README
> >
> > ...I see every .svn/entries the entire tree being opened.
>
> Presumably we're locking the anchor recursively.
>
> I thought we had fixed this ages ago, but I guess not.
>
Without being anything near an WC hacker, it looks like you're right.
Would just checking if the entry is a file and using depth 0 in that case
be enough? I'll prepare a patch...
Regards,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PATCH] Re: slow status
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
On Sun, 10 Oct 2004, Greg Hudson wrote:
> On Sun, 2004-10-10 at 12:04, Greg Hudson wrote:
> > I think doing that can also simplify the logic. If there is a target,
> > lock the anchor non-recursively and then lock the target (recursively or
> > not, as specified). If there is no target, lock the anchor recursively
> > or not as specified. That's fewer cases than your patch has.
>
> That was imprecise and missed the subtlety that we need to lock the
> subdirs for a non-recursive status. But there's still no need to
> consider more cases than I specified. So, to be more precise:
>
> If there is a separate anchor and target:
> Lock the anchor with depth 0
> Lock the target with depth (recursive ? -1 : 1)
> Else
> Lock the anchor with depth (recursive ? -1 : 1)
>
> No need to consider whether the target is a file or anything like that.
>
>
Hmmm... Seems like I have to check and only lock target if it is
non-empty *and* a directory. Locking a file in a locked directory gives an
wc locked error.
Thanks,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PATCH] Re: slow status, next next version
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
On Mon, 25 Oct 2004, Barry Scott wrote:
> Am I right in assuming that this is still not fixed 1.1.1?
>
> I see 4.6 seconds to do svn st -N of the top of the 1.1.1 tag in my
> working copy.
> Let me ask a simple question: Why do you need to lock at all to get a
> status?
> If locking is a problem stop doing it :-)
>
Should be fixed in r11439, but it was fixed rather late in the 1.1.1 cycle
and I didn't want to include more stuff in 1.1.1 at that time. It owuld be
nice to know, however, if it makes any different if you could try the
patch out.
Regards,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PATCH] Re: slow status, next next version
Posted by Barry Scott <ba...@barrys-emacs.org>.
Am I right in assuming that this is still not fixed 1.1.1?
I see 4.6 seconds to do svn st -N of the top of the 1.1.1 tag in my
working copy.
After its in the windows cache its faster.
Let me ask a simple question: Why do you need to lock at all to get a
status?
If locking is a problem stop doing it :-)
Barry
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PATCH] Re: slow status, next next version
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
On Tue, 12 Oct 2004, Peter N. Lundblad wrote:
> On Tue, 12 Oct 2004, Peter N. Lundblad wrote:
>
> > Need to take are of the update case further down in the function as well.
> > Here's another try. Passes tests over ra_local. It wasn't as simple as you
> > stated, but I hope it is correct now.
> >
> And of course I hit send a little too early. It breaks a test. Will send
> another one. Sorry.
>
Here we go again. It actually passes the tests this time. I replaced a
wc_adm_retrieve with wc_adm_probe_retrieve...
Re: [PATCH] Re: slow status, next version
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
On Tue, 12 Oct 2004, Peter N. Lundblad wrote:
> Need to take are of the update case further down in the function as well.
> Here's another try. Passes tests over ra_local. It wasn't as simple as you
> stated, but I hope it is correct now.
>
> Thanks for this and many other reviews, Greg.
>
And of course I hit send a little too early. It breaks a test. Will send
another one. Sorry.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PATCH] Re: slow status, next version
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
On Sun, 10 Oct 2004, Greg Hudson wrote:
> If there is a separate anchor and target:
> Lock the anchor with depth 0
> Lock the target with depth (recursive ? -1 : 1)
> Else
> Lock the anchor with depth (recursive ? -1 : 1)
>
Need to take are of the update case further down in the function as well.
Here's another try. Passes tests over ra_local. It wasn't as simple as you
stated, but I hope it is correct now.
Thanks for this and many other reviews, Greg.
Regards,
//Peter
Re: [PATCH] Re: slow status
Posted by Greg Hudson <gh...@MIT.EDU>.
On Sun, 2004-10-10 at 12:04, Greg Hudson wrote:
> I think doing that can also simplify the logic. If there is a target,
> lock the anchor non-recursively and then lock the target (recursively or
> not, as specified). If there is no target, lock the anchor recursively
> or not as specified. That's fewer cases than your patch has.
That was imprecise and missed the subtlety that we need to lock the
subdirs for a non-recursive status. But there's still no need to
consider more cases than I specified. So, to be more precise:
If there is a separate anchor and target:
Lock the anchor with depth 0
Lock the target with depth (recursive ? -1 : 1)
Else
Lock the anchor with depth (recursive ? -1 : 1)
No need to consider whether the target is a file or anything like that.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PATCH] Re: slow status
Posted by Greg Hudson <gh...@MIT.EDU>.
On Fri, 2004-10-08 at 15:19, Peter N. Lundblad wrote:
> On Thu, 7 Oct 2004, Greg Hudson wrote:
>
> > Presumably we're locking the anchor recursively.
> >
> > I thought we had fixed this ages ago, but I guess not.
> >
> Maybe the attached patch will fix this problem?
Locking the anchor with depth 2 is a botch. If you look at
svn_client_switch(), you can see an example of locking the anchor
non-recursively and then locking the target.
I think doing that can also simplify the logic. If there is a target,
lock the anchor non-recursively and then lock the target (recursively or
not, as specified). If there is no target, lock the anchor recursively
or not as specified. That's fewer cases than your patch has.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
[PATCH] Re: slow status
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
On Thu, 7 Oct 2004, Greg Hudson wrote:
> Presumably we're locking the anchor recursively.
>
> I thought we had fixed this ages ago, but I guess not.
>
Maybe the attached patch will fix this problem?
Re: slow status
Posted by kf...@collab.net.
Greg Hudson <gh...@MIT.EDU> writes:
> Presumably we're locking the anchor recursively.
>
> I thought we had fixed this ages ago, but I guess not.
So did I. I responded to the original reporter on the users@ list,
saying:
> Is your client (as well as the server side) also 1.1.0? I would have
> thought this was
>
> http://subversion.tigris.org/issues/show_bug.cgi?id=1680
>
> ...but that issue is marked as resolved in 1.1.0.
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: slow status
Posted by Greg Hudson <gh...@MIT.EDU>.
On Thu, 2004-10-07 at 19:57, Ben Collins-Sussman wrote:
> $ svn status subversion/libsvn_fs_base/bdb/dbt.c
>
> Then I only see the bdb/.svn/entries file being opened.
>
> But when I run
>
> $ svn status README
>
> ...I see every .svn/entries the entire tree being opened.
Presumably we're locking the anchor recursively.
I thought we had fixed this ages ago, but I guess not.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: slow status
Posted by Ben Collins-Sussman <su...@collab.net>.
On Oct 7, 2004, at 5:37 PM, John Williams wrote:
> I am trying to put a large website project into subversion-1.1.0.
> fsfs is a big improvement, but there are still scalability problems.
>
> Here is an especially obvious one.
>
> $ date; svn status newsite.html ; date
> Thu Oct 7 16:28:23 MDT 2004
> Thu Oct 7 16:30:34 MDT 2004
>
> 2 minutes(!) for an svn status command on a single file.
> I am using svnserve, but that doesn't matter.
>
> An strace on the process shows that svn status is reading *every*
> .svn/format and .svn/entries file in the entire project, starting at .
> (the current directory), even though I only asked for the status of a
> single regular file.
>
What the...??
I can confirm this too. When I run an OSX 'ktrace' on
$ svn status subversion/libsvn_fs_base/bdb/dbt.c
Then I only see the bdb/.svn/entries file being opened.
But when I run
$ svn status README
...I see every .svn/entries the entire tree being opened.
Developer-folk, is this expected?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org