Re-use connection for svn:externals

[Phillip Hellewell <ss...@gmail.com>]

Hi,

I'm not sure if this is related to issue 1448 or not (it's kinda like the
opposite of 1448 actually), but I use externals extensively and doing an
update is slow.  It appears to be creating a new connection for each
external even though it's the same repository on the same server.  Each
connection takes about 3 seconds to make.

I have a large project (over 10,000 files) that takes about 10 seconds to do
an update (with no externals).  Then I have a sibling folder that acts as a
"module" to provide a limited view of the project by using relative
externals to about a dozen subdirs of the project.  This "module" folder
takes 40 seconds to do an update.  So it takes longer to update this
"module" folder even though it only contains a subset of the project!  Even
individual file externals take 3 seconds each to update.

Since all the externals are relative, they are to the same repository on the
same server, so can't we re-use existing connection?

Thanks,
Phillip

P.S.  I am connecting to a server over the Internet, using HTTPS with
Kerberos authentication.

Re: Re-use connection for svn:externals

[Ivan Zhakov <iv...@visualsvn.com>]

On Tue, Feb 16, 2010 at 9:18 AM, Phillip Hellewell <ss...@gmail.com> wrote:
> Alright.  Here are my thoughts so far based on what others have said:
>
> I'm not quite sure about Justin's suggestion, because I don't think the
> delay is in establishing the TCP connection; I think it is in performing the
> authentication and stuff.  So just re-using a TCP connection won't buy us
> much, right?
>
Sharing TCP connection has problems in case of connection based
authentication/multithreading and etc.

> As for Bert's concern about always opening at the repository root could
> cause issues like #3242, does everyone agree that we should avoid doing that
> as part of the solution?
>
> So could we (as some suggested) take an existing RA session and reparent it
> to the root?  Then if using that doesn't work, we fall back to creating a
> new session (since as Stefan suggested, this is an optimization so it
> doesn't need to break anything).
>
Yes, you can avoid issues like #3242 if you open RA session to URL and
then reparent to the root.


-- 
Ivan Zhakov
VisualSVN Team

Re: Re-use connection for svn:externals

[Justin Erenkrantz <ju...@erenkrantz.com>]

On Tue, Feb 16, 2010 at 12:33 AM, Ivan Zhakov <iv...@visualsvn.com> wrote:
> You can do that only if you assume that connections from different
> sessions should have same authentication, which is not true imho.

If it's pointing to the same server, wouldn't it?

How often is it that you would do *connection-level* auth (like
Kerberos) and have different creds for different paths?

Even if it were different, the failure path would be handled just like
any auth failure - ie would need to (re)present new creds in response
to a 401 auth failure...  -- justin

Re: Re-use connection for svn:externals

[Ivan Zhakov <iv...@visualsvn.com>]

On Tue, Feb 16, 2010 at 10:10 AM, Justin Erenkrantz
<ju...@erenkrantz.com> wrote:
> On Mon, Feb 15, 2010 at 10:18 PM, Phillip Hellewell <ss...@gmail.com> wrote:
>> Alright.  Here are my thoughts so far based on what others have said:
>>
>> I'm not quite sure about Justin's suggestion, because I don't think the
>> delay is in establishing the TCP connection; I think it is in performing the
>> authentication and stuff.  So just re-using a TCP connection won't buy us
>> much, right?
>
> If you are using Kerberos-based auth and you've already completed
> authentication on that connection, you likely won't need to re-auth if
> you reuse the TCP connection via Serf or Neon.  (All hail stupid
> authentication systems like Kerberos which are connection-oriented.
> Not.)  -- justin
>
You can do that only if you assume that connections from different
sessions should have same authentication, which is not true imho.


-- 
Ivan Zhakov
VisualSVN Team

Re: Re-use connection for svn:externals

[Justin Erenkrantz <ju...@erenkrantz.com>]

On Mon, Feb 15, 2010 at 10:18 PM, Phillip Hellewell <ss...@gmail.com> wrote:
> Alright.  Here are my thoughts so far based on what others have said:
>
> I'm not quite sure about Justin's suggestion, because I don't think the
> delay is in establishing the TCP connection; I think it is in performing the
> authentication and stuff.  So just re-using a TCP connection won't buy us
> much, right?

If you are using Kerberos-based auth and you've already completed
authentication on that connection, you likely won't need to re-auth if
you reuse the TCP connection via Serf or Neon.  (All hail stupid
authentication systems like Kerberos which are connection-oriented.
Not.)  -- justin

Re: Re-use connection for svn:externals

[Phillip Hellewell <ss...@gmail.com>]

Alright.  Here are my thoughts so far based on what others have said:

I'm not quite sure about Justin's suggestion, because I don't think the
delay is in establishing the TCP connection; I think it is in performing the
authentication and stuff.  So just re-using a TCP connection won't buy us
much, right?

As for Bert's concern about always opening at the repository root could
cause issues like #3242, does everyone agree that we should avoid doing that
as part of the solution?

So could we (as some suggested) take an existing RA session and reparent it
to the root?  Then if using that doesn't work, we fall back to creating a
new session (since as Stefan suggested, this is an optimization so it
doesn't need to break anything).

Do I understand correctly so far?

Phillip

On Mon, Feb 15, 2010 at 3:18 AM, Julian Foad <ju...@wandisco.com>wrote:

> On Tue, 2010-02-09, Phillip Hellewell wrote:
> > On Tue, Feb 9, 2010 at 10:09 AM, Julian Foad <julian.foad@wandisco.com
> >wrote:
> >
> > >
> > > Would you, or anyone you know, be interested in working on it? I would
> > > be glad to give you some help and guidance.
> > >
> >
> > I don't have any experience with the subversion source code right now,
> but
> > I'd be willing to become familiar with it if no one else were interested
> in
> > implementing it.
> >
> > But based on the lively discussion I see so far, it sounds like a feature
> > other people really want to, so I'm guessing you or Ivan or C. Michael
> will
> > probably beat me to it.
>
> Not terribly likely. While I'm interested, there is more urgent stuff
> that I'm working on, and I expect the others are in the same position.
> So please do jump in and get the ball rolling by posting your first
> thought or question on how to do it.
>
> - Julian
>
>
>

Re: Re-use connection for svn:externals

[Julian Foad <ju...@wandisco.com>]

On Tue, 2010-02-09, Phillip Hellewell wrote:
> On Tue, Feb 9, 2010 at 10:09 AM, Julian Foad <ju...@wandisco.com>wrote:
> 
> >
> > Would you, or anyone you know, be interested in working on it? I would
> > be glad to give you some help and guidance.
> >
> 
> I don't have any experience with the subversion source code right now, but
> I'd be willing to become familiar with it if no one else were interested in
> implementing it.
> 
> But based on the lively discussion I see so far, it sounds like a feature
> other people really want to, so I'm guessing you or Ivan or C. Michael will
> probably beat me to it.

Not terribly likely. While I'm interested, there is more urgent stuff
that I'm working on, and I expect the others are in the same position.
So please do jump in and get the ball rolling by posting your first
thought or question on how to do it.

- Julian

Re: Re-use connection for svn:externals

[Phillip Hellewell <ss...@gmail.com>]

On Tue, Feb 9, 2010 at 10:09 AM, Julian Foad <ju...@wandisco.com>wrote:

>
> Would you, or anyone you know, be interested in working on it? I would
> be glad to give you some help and guidance.
>

I don't have any experience with the subversion source code right now, but
I'd be willing to become familiar with it if no one else were interested in
implementing it.

But based on the lively discussion I see so far, it sounds like a feature
other people really want to, so I'm guessing you or Ivan or C. Michael will
probably beat me to it.

Phillip

Re: Re-use connection for svn:externals

[Ivan Zhakov <iv...@visualsvn.com>]

On Tue, Feb 9, 2010 at 7:51 PM, C. Michael Pilato <cm...@collab.net> wrote:
> Ivan Zhakov wrote:
>> On Tue, Feb 9, 2010 at 7:09 PM, Julian Foad <ju...@wandisco.com> wrote:
>>> There are several places where the client should/could re-use a
>>> connection. During a multi-target update command like "svn up a b c" is
>>> another example.
>>>
>>> Would you, or anyone you know, be interested in working on it? I would
>>> be glad to give you some help and guidance.
>>>
>> Agreed. svn_client_* functions spend a lot time on creating RA
>> connection. My idea was to introduce "connection pool" to hold and
>> reuse RA connections. This pool can be stored in svn_client_ctx_t or
>> in separate object.
>
> I was about to suggest exactly the same idea, Ivan!  I'd love to have such a
> thing handy with some nice private APIs for locating a connection for use
> (or re-use).
The only problem I see is RA session root relative paths [1], since we
end up with pool of connection for one repository but with different
session root. We may workaround it using convention that svn_client_*
RA sessions always rooted to repository root [2].

> I imagine that anyone who has looked at the merge tracking
> code would see the value here.
Yes, merge tracking code is what I was thinking about.

[1] http://svn.haxx.se/dev/archive-2009-10/0334.shtml
[2] http://svn.haxx.se/dev/archive-2009-10/0315.shtml

-- 
Ivan Zhakov
VisualSVN Team

Re: Re-use connection for svn:externals

["C. Michael Pilato" <cm...@collab.net>]

Ivan Zhakov wrote:
> On Tue, Feb 9, 2010 at 7:09 PM, Julian Foad <ju...@wandisco.com> wrote:
>> There are several places where the client should/could re-use a
>> connection. During a multi-target update command like "svn up a b c" is
>> another example.
>>
>> Would you, or anyone you know, be interested in working on it? I would
>> be glad to give you some help and guidance.
>>
> Agreed. svn_client_* functions spend a lot time on creating RA
> connection. My idea was to introduce "connection pool" to hold and
> reuse RA connections. This pool can be stored in svn_client_ctx_t or
> in separate object.

I was about to suggest exactly the same idea, Ivan!  I'd love to have such a
thing handy with some nice private APIs for locating a connection for use
(or re-use).  I imagine that anyone who has looked at the merge tracking
code would see the value here.

-- 
C. Michael Pilato <cm...@collab.net>
CollabNet   <>   www.collab.net   <>   Distributed Development On Demand

Re: Re-use connection for svn:externals

[Ivan Zhakov <iv...@visualsvn.com>]

On Tue, Feb 9, 2010 at 7:09 PM, Julian Foad <ju...@wandisco.com> wrote:
> On Mon, 2010-02-08, Phillip Hellewell wrote:
>> I'm not sure if this is related to issue 1448 or not (it's kinda like the
>> opposite of 1448 actually), but I use externals extensively and doing an
>> update is slow.  It appears to be creating a new connection for each
>> external even though it's the same repository on the same server.  Each
>> connection takes about 3 seconds to make.
>>
>> I have a large project (over 10,000 files) that takes about 10 seconds to do
>> an update (with no externals).  Then I have a sibling folder that acts as a
>> "module" to provide a limited view of the project by using relative
>> externals to about a dozen subdirs of the project.  This "module" folder
>> takes 40 seconds to do an update.  So it takes longer to update this
>> "module" folder even though it only contains a subset of the project!  Even
>> individual file externals take 3 seconds each to update.
>>
>> Since all the externals are relative, they are to the same repository on the
>> same server, so can't we re-use existing connection?
>
> Yes, that would be a good enhancement.
>
> There are several places where the client should/could re-use a
> connection. During a multi-target update command like "svn up a b c" is
> another example.
>
> Would you, or anyone you know, be interested in working on it? I would
> be glad to give you some help and guidance.
>
Agreed. svn_client_* functions spend a lot time on creating RA
connection. My idea was to introduce "connection pool" to hold and
reuse RA connections. This pool can be stored in svn_client_ctx_t or
in separate object.


-- 
Ivan Zhakov
VisualSVN Team

Re: Re-use connection for svn:externals

[Julian Foad <ju...@wandisco.com>]

On Mon, 2010-02-08, Phillip Hellewell wrote:
> I'm not sure if this is related to issue 1448 or not (it's kinda like the
> opposite of 1448 actually), but I use externals extensively and doing an
> update is slow.  It appears to be creating a new connection for each
> external even though it's the same repository on the same server.  Each
> connection takes about 3 seconds to make.
> 
> I have a large project (over 10,000 files) that takes about 10 seconds to do
> an update (with no externals).  Then I have a sibling folder that acts as a
> "module" to provide a limited view of the project by using relative
> externals to about a dozen subdirs of the project.  This "module" folder
> takes 40 seconds to do an update.  So it takes longer to update this
> "module" folder even though it only contains a subset of the project!  Even
> individual file externals take 3 seconds each to update.
> 
> Since all the externals are relative, they are to the same repository on the
> same server, so can't we re-use existing connection?

Yes, that would be a good enhancement.

There are several places where the client should/could re-use a
connection. During a multi-target update command like "svn up a b c" is
another example.

Would you, or anyone you know, be interested in working on it? I would
be glad to give you some help and guidance.

- Julian