You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Justin Erenkrantz <je...@apache.org> on 2002/07/10 10:05:39 UTC
Speeding up the 'svn commit' process?
My current gripe is that 'svn commit' just takes too long to pop
up an editor. This seems to be related to the efficiency of
svn_client__harvest_committables function. It is taking me about
10-15 minutes for my editor to pop up with the list of files that
are changed. My starting point is a linux-2.4.14 kernel tree and
then I'm trying to upgrade to 2.4.15, 2.4.16, 2.4.17, etc.
As I know this is something that has been brought up before, but
boy are we reading the .entries files too much. It currently
looks to be a 2:1 ratio. For a directory with about 512 files,
SVN parses the .entries file about 996 times. IMHO, that's about
995 times more than it needs to. For each time it parses the
.entries file, it reloads and reparses the XML file.
So, what would we have to do to keep a single copy of the parsed
entries file and pass it around everywhere? Should I just start
trying to add entry to functions and go from there? I believe
the second entries parsing comes from svn_wc_props_modified_p -
which has the distinction of not wanting info on deleted files.
I think we can work around that even if we have the complete
entries for that directory (no need to re-read it).
Thoughts? It's obvious people have thought about this before
and gave up. So, before I start down this road, I'd like to
get feedback from those who tried before. =) -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Speeding up the 'svn commit' process?
Posted by Ben Collins-Sussman <su...@collab.net>.
Justin Erenkrantz <je...@apache.org> writes:
> My current gripe is that 'svn commit' just takes too long to pop
> up an editor. This seems to be related to the efficiency of
> svn_client__harvest_committables function. It is taking me about
> 10-15 minutes for my editor to pop up with the list of files that
> are changed. My starting point is a linux-2.4.14 kernel tree and
> then I'm trying to upgrade to 2.4.15, 2.4.16, 2.4.17, etc.
I'm thrilled that we now have an army of programmers discussing how to
make libsvn_wc cache entries. Once this problem is tackled, it will
not only massively speed up commits, but status and update as well.
All three operations involve huge tree walks.
The key here is to do it in small steps... rather than any one person
take on a herculean rewrite in solitary confinement. I think everyone
realizes this now. It looks like Philip has already committed a good
first step; keep the discussion going!
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Speeding up the 'svn commit' process?
Posted by Kevin Pilch-Bisson <ke...@pilch-bisson.net>.
On Thu, Jul 11, 2002 at 04:08:45PM -0400, Greg Hudson wrote:
> On Wed, 2002-07-10 at 09:09, Kevin Pilch-Bisson wrote:
> > I disagree. Rather, I think we should export the baton carrying functions, as
> > well as two functions to create/finish with batons. Then the consumer of the
> > API determines the life of the baton. This makes sense, since they are the
> > only ones who know how long the operation is going to last.
>
> Uh, if I understand what you're proposing, this is not really a baton as
> we understand it.
>
> First, it shouldn't be a void *. There's only one wc library, and thus
> only one structure type needed to hold cached data. If you want to use
> forward structure declarations to make it opaque to the caller, great,
> but don't make it a void *. No need to sacrifice type safety
> unnecessarily.
You (and Karl and Philip, and everyone else) are right. There isn't (aren't?)
going to be more than one type of working copy cache, so it shouldn't be void
*. I do think it should be opaque. However, as that change has already been
comitted, the point is moot.
>
> Second, as Ben said in a separate thread, when most of us see "baton" we
> think "user data"--that is, something opaque to the callee but
> transparent to the caller. (The editor interface may be an exception,
> but once you're in vtable-land things are already kind of convoluted.)
> What we have here is more of a "context"--something opaque to the caller
> but transparent to the callee.
>
Ah yes, it was the editor interface thing that was confusing me. I'll second
context as a good term for this type of structure.
And, I'll try to be more precise about what I'm saying in the future. :)
--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Kevin Pilch-Bisson http://www.pilch-bisson.net
"Historically speaking, the presences of wheels in Unix
has never precluded their reinvention." - Larry Wall
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Re: Speeding up the 'svn commit' process?
Posted by Greg Hudson <gh...@MIT.EDU>.
On Wed, 2002-07-10 at 09:09, Kevin Pilch-Bisson wrote:
> I disagree. Rather, I think we should export the baton carrying functions, as
> well as two functions to create/finish with batons. Then the consumer of the
> API determines the life of the baton. This makes sense, since they are the
> only ones who know how long the operation is going to last.
Uh, if I understand what you're proposing, this is not really a baton as
we understand it.
First, it shouldn't be a void *. There's only one wc library, and thus
only one structure type needed to hold cached data. If you want to use
forward structure declarations to make it opaque to the caller, great,
but don't make it a void *. No need to sacrifice type safety
unnecessarily.
Second, as Ben said in a separate thread, when most of us see "baton" we
think "user data"--that is, something opaque to the callee but
transparent to the caller. (The editor interface may be an exception,
but once you're in vtable-land things are already kind of convoluted.)
What we have here is more of a "context"--something opaque to the caller
but transparent to the callee.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Speeding up the 'svn commit' process?
Posted by mark benedetto king <bk...@inquira.com>.
On Wed, Jul 10, 2002 at 09:09:51AM -0400, Kevin Pilch-Bisson wrote:
> On Wed, Jul 10, 2002 at 08:55:50AM -0400, mark benedetto king wrote:
> > 2.) The library consumes its own API. While there's nothing wrong with
> > this, it does mean that nearly all of the exported functions would have
> > needed baton-ized versions. Take a look. There are a whole bunch of
> > exported functions.
> >
> > My feeling is that libsvn_wc needs to be split into two layers: one that
> > is baton-only, and another that consumes that one's API. This should
> > resolve (2), and make it easier to focus on (1). This will be a big
> > effort, but the payoffs could be huge. I measured, IIRC, over 90% of
> > client-side processing spent in manipulating the entries files.
> >
> I disagree. Rather, I think we should export the baton carrying functions, as
> well as two functions to create/finish with batons. Then the consumer of the
> API determines the life of the baton. This makes sense, since they are the
> only ones who know how long the operation is going to last.
Once we have a baton-driven API, we can look at the scope of pushing the
new API onto the existing libsvn_wc consumers. I didn't look at this, I
was trying to keep the change's scope down. It's possible that it would
be less work overall to push the API changes as well, rather than build
the "batonification" layer.
--ben
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Speeding up the 'svn commit' process?
Posted by Kevin Pilch-Bisson <ke...@pilch-bisson.net>.
On Wed, Jul 10, 2002 at 08:55:50AM -0400, mark benedetto king wrote:
> 2.) The library consumes its own API. While there's nothing wrong with
> this, it does mean that nearly all of the exported functions would have
> needed baton-ized versions. Take a look. There are a whole bunch of
> exported functions.
>
> My feeling is that libsvn_wc needs to be split into two layers: one that
> is baton-only, and another that consumes that one's API. This should
> resolve (2), and make it easier to focus on (1). This will be a big
> effort, but the payoffs could be huge. I measured, IIRC, over 90% of
> client-side processing spent in manipulating the entries files.
>
I disagree. Rather, I think we should export the baton carrying functions, as
well as two functions to create/finish with batons. Then the consumer of the
API determines the life of the baton. This makes sense, since they are the
only ones who know how long the operation is going to last.
--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Kevin Pilch-Bisson http://www.pilch-bisson.net
"Historically speaking, the presences of wheels in Unix
has never precluded their reinvention." - Larry Wall
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Re: Speeding up the 'svn commit' process?
Posted by mark benedetto king <bk...@inquira.com>.
On Wed, Jul 10, 2002 at 03:05:39AM -0700, Justin Erenkrantz wrote:
> Thoughts? It's obvious people have thought about this before
> and gave up. So, before I start down this road, I'd like to
> get feedback from those who tried before. =) -- justin
>
MOHIAM: Many men have tried.
PAUL: They tried and failed?
MOHIAM: They tried and died.
I spent a little over half a day looking at making libsvn_wc interally
baton driven without changing the exposed API. It quickly blossomed
into a huge effort. Had it only been a lot of simple changes, I might
have persevered, but there are some fine points that scared me off:
1.) There are really two views of the entries: one with deleted entries
shown and one with them ignored. It's not impossible to effect this with
an in-memory cache, but it means that rework needs to be especially careful,
because it needs to keep track of which view the code is interested in.
This probably also means that a "delete" of an entry needs to remove it from
the "ignore deleted" view immediately, but not from the other view until
later. I didn't look closely at this, but it worried me.
2.) The library consumes its own API. While there's nothing wrong with
this, it does mean that nearly all of the exported functions would have
needed baton-ized versions. Take a look. There are a whole bunch of
exported functions.
My feeling is that libsvn_wc needs to be split into two layers: one that
is baton-only, and another that consumes that one's API. This should
resolve (2), and make it easier to focus on (1). This will be a big
effort, but the payoffs could be huge. I measured, IIRC, over 90% of
client-side processing spent in manipulating the entries files.
--ben
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Speeding up the 'svn commit' process?
Posted by Philip Martin <ph...@codematters.co.uk>.
Justin Erenkrantz <je...@apache.org> writes:
> On Wed, Jul 10, 2002 at 11:46:18AM +0100, Philip Martin wrote:
> > I've started looking at issue #749
> >
> > http://subversion.tigris.org/issues/show_bug.cgi?id=749
> >
> > This will allow some cacheing of the entries.
>
> Hey, whaddaya know. =)
>
> So, just to make sure we're on the same wavelength, when we
> create this baton, we should be performing svn_wc_entries_read()
> and storing a copy of that in the baton. Perhaps going through
> and creating a duplicate hashtable that has the deleted items
> removed - still no need for a reparse of the entries file.
>
> Then, any function that accesses the wc (say svn_wc_entry,
> svn_wc_props_modified_p, etc, will require this baton). They
> can then use baton->entries to get their entry.
>
> Am I close? -- justin
Yes. Initially there will simply be a requirement to pass a baton
into those functions that modify the adminstrative area, thus ensuring
that all modifications are using the log file mechanism. I'm working
on introducing the baton without altering very much else, only making
changes that are necessary to get the correct locking/loggy behaviour.
Once the locking works, adding cacheing to the baton should be
possible.
--
Philip Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Speeding up the 'svn commit' process?
Posted by Justin Erenkrantz <je...@apache.org>.
On Wed, Jul 10, 2002 at 11:46:18AM +0100, Philip Martin wrote:
> I've started looking at issue #749
>
> http://subversion.tigris.org/issues/show_bug.cgi?id=749
>
> This will allow some cacheing of the entries.
Hey, whaddaya know. =)
So, just to make sure we're on the same wavelength, when we
create this baton, we should be performing svn_wc_entries_read()
and storing a copy of that in the baton. Perhaps going through
and creating a duplicate hashtable that has the deleted items
removed - still no need for a reparse of the entries file.
Then, any function that accesses the wc (say svn_wc_entry,
svn_wc_props_modified_p, etc, will require this baton). They
can then use baton->entries to get their entry.
Am I close? -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Speeding up the 'svn commit' process?
Posted by Philip Martin <ph...@codematters.co.uk>.
Justin Erenkrantz <je...@apache.org> writes:
> So, what would we have to do to keep a single copy of the parsed
> entries file and pass it around everywhere? Should I just start
> trying to add entry to functions and go from there? I believe
> the second entries parsing comes from svn_wc_props_modified_p -
> which has the distinction of not wanting info on deleted files.
> I think we can work around that even if we have the complete
> entries for that directory (no need to re-read it).
I've started looking at issue #749
http://subversion.tigris.org/issues/show_bug.cgi?id=749
This will allow some cacheing of the entries.
--
Philip Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org