You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Niel De Clerk <ND...@Angloplat.com> on 2008/01/04 05:09:11 UTC
Faster SVN Status
Good day
We use SVN over a large distributed network that's normally slow and
often unreliable. It has changed our lives drastically though -
excellent product!
The files that we store in the repository have various types and most of
them remain unchanged (but it is nice having them there for initial
deployment to new clients).
The files in the repository that change regularly all have a .kb
extension, and the applications running on our clients are mostly
concerned with these files.
Is it possible to do a "svn status -verbose -show-updates" that will
only list the *.kb files (including the ones in sub-directories)? I.e.
is there a way (direct or workaround) to include a --targets switch for
a status command? Or a way to ignore files (versioned or unversioned)
when a certain filter/mask is provided?
Will this improve the speed of the command execution? (Note this will
reduce the number of files to consider by more than half)
I've tried to the following syntax c:\myfiles\*.kb, but this seems to do
a separate transaction for each file and is much slower.
Many thanks
Niel
This email is subject to our email legal notice, to view browse to http://www.angloplatinum.co.za/E-mailLegalNotice/
Re: Faster SVN Status
Posted by Les Mikesell <le...@gmail.com>.
Steven Bakke wrote:
>>> However through profiling we found a performance bottleneck in that
>>> it starts a new repository session (network connection) once for each
>>> target of the command. In our case the problem was with 'svn
>>> update', but I think it applies to almost anything. The performance
>>> became dominated by ~2-3 seconds per target overhead each time it
>>> made a new svn+ssh connection to the server.
>>
>> What can possibly cause a several-second delay? Does your reverse DNS
>> lookup resolve quickly? Are IDENT queries dropped and timing out?
>>
>
> I don't know for sure. First, there is the initiation of an SSH
> connection. That is followed by creation of an ra_session. I'm
> wondering if we could do a comparision by using the file:// access
> method. I'm not the person who did the code profiling.
Some overhead in starting a new ssh session would be normal, but several
seconds is an enormous number of CPU operations these days so I'd look
elsewhere for this kind of delay. As you connect, sshd is probably doing
a reverse DNS lookup on the connecting IP address and an IDENT query to
the connecting client for the user owning the socket, and the time for
those reponses (which may have nothing to do with the svn server itself)
may dominate the elapsed time you measure. Getting a big improvement
might be as simple as turning on a local caching nameserver or changing
logging options.
--
Les Mikesell
lesmikesell@gmail.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Faster SVN Status
Posted by Steven Bakke <st...@amd.com>.
On Jan 4, 2008, at 10:42 AM, Les Mikesell wrote:
> Steven Bakke wrote:
>> However through profiling we found a performance bottleneck in that
>> it starts a new repository session (network connection) once for
>> each target of the command. In our case the problem was with 'svn
>> update', but I think it applies to almost anything. The
>> performance became dominated by ~2-3 seconds per target overhead
>> each time it made a new svn+ssh connection to the server.
>
> What can possibly cause a several-second delay? Does your reverse
> DNS lookup resolve quickly? Are IDENT queries dropped and timing out?
>
I don't know for sure. First, there is the initiation of an SSH
connection. That is followed by creation of an ra_session. I'm
wondering if we could do a comparision by using the file:// access
method. I'm not the person who did the code profiling.
-steve
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Faster SVN Status
Posted by Steven Bakke <st...@amd.com>.
On Jan 7, 2008, at 7:00 AM, Niel De Clerk wrote:
> Agreed - but that does explain why the first is faster than the
> second,
> even though you don't get more information from the second - I can't
> think of a reason why you'd want to use the second option rather than
> the first, except if you're interested in only about 10% of the
> files in
> the directory.
This is what I was attempting to explain in my original response.
When we profiled the code we found that when you do the following:
% svn <svn-command> file1 file2 file3 ...
this will initiate a separate connection for each target of the
command. (which is what happens when you
use shell wildcards. It's completely unnecessary, but this happens
to be
the way that the svn client is written. Similarly, when you give it a
single
directory as a target, it will run that command using a single
connection. Recursive
operations are done using that same connection.
In hacking with the subversion source code to force it to use one
connection per svn
command invocation, we got a very significant speedup.
>
>
> On a different note, is there a way to include subdirectories when
> using
> wild-cards? I.e. use something like *.kb to include all files in the
> current directory and all of its subdirectories?
>
> Cheers
> Niel
> -----Original Message-----
> From: Andy Levy [mailto:andy.levy@gmail.com]
> Sent: 07 January 2008 13:55
> To: Niel De Clerk
> Cc: users@subversion.tigris.org
> Subject: Re: RE: Re: Faster SVN Status
>
> On Jan 7, 2008 12:21 AM, Niel De Clerk <ND...@angloplat.com> wrote:
>> I definitely also saw the additional delay introduced with multiple
>> transactions. Below are 2 ways of getting exactly the same thing.
>> However the first give all the results followed by a single "Status
>> against revision X" message where the second will give the "Status
>> against revision X" for each individual file. To me this indicates
> some
>> additional handshaking? When on the same network as the repository,
>> this is a non issue, but when on a remote connection (e.g. using VPN
>> from home) this can make the difference between a 30 second and a 190
>> second transaction.
>>
>> Svn status --show-updates --verbose -N c:\my-files
>
> This asks for the status of the directory. One transaction.
>
>> Svn status --show-updates --verbose c:\my-files\*.*
>
> This asks for the status of each child of the directory because the
> shell expands the wildcards. It's the same as if you were to run:
>
> svn status --show-updates --verbose c:\my-files\file1
> c:\my-files\file2 c:\my-files\file3 c:\my-files\directory1
>
> It's not (IMHO) "exactly the same thing." You're asking for 2
> different kinds of information.
>
>>
>> -----Original Message-----
>> From: news [mailto:news@ger.gmane.org] On Behalf Of Vincent
>> Tondellier
>> Sent: 06 January 2008 01:51
>> To: users@subversion.tigris.org
>> Subject: RE: Re: Faster SVN Status
>>
>> Harvey, Edward wrote:
>>
>>> I can't say exactly what's happening in Niel's situation, but I
>> support
>>> a company with worldwide network, and when we SSH from Boston to
> India
>>> or vice-versa, there is a few second delay. Because there's a 250ms
>>> ping response delay, and the initial ssh handshake is the equivalent
>> of
>>> 10-15 small packet round-trips, it's about 5-6 seconds before the
>>> command prompt appears.
>>>
>>> If you use svn:// protocol and svnserve, there's just the basic tcp
>>> handshake before useful data starts going across, so it's much
> faster.
>>> Call it 1-2 x the ping delay. About a half a second for me.
>>>
>>> If you use https://, it looks like something like half a dozen small
>>> packet round trips. In my case, about 2 seconds delay. So it's
>> faster
>>> than svn+ssh, but not as fast as svnserve.
>>>
>>
>> You should try the connection caching feature of OpenSSH, it only
>> establish
>> one master ssh connection, and opens new ssh "channels" for each new
>> connection you make. The GCC wiki have some explanations on this :
>>
>> http://gcc.gnu.org/wiki/SSH connection caching
>>
>>>> -----Original Message-----
>>>> From: Les Mikesell [mailto:lesmikesell@gmail.com]
>>>> Sent: Friday, January 04, 2008 10:42 AM
>>>> To: Steven Bakke
>>>> Cc: Niel De Clerk; users@subversion.tigris.org
>>>> Subject: Re: Faster SVN Status
>>>>
>>>> Steven Bakke wrote:
>>>>>
>>>>> However through profiling we found a performance bottleneck in
> that
>>>> it
>>>>> starts a new repository session (network connection) once for
> each
>>>>> target of the command. In our case the problem was with 'svn
>>>> update',
>>>>> but I think it applies to almost anything. The performance
> became
>>>>> dominated by ~2-3 seconds per target overhead each time it made a
>>> new
>>>>> svn+ssh connection to the server.
>>>>
>>>> What can possibly cause a several-second delay? Does your reverse
>> DNS
>>>> lookup resolve quickly? Are IDENT queries dropped and timing out?
>>>>
>>>> --
>>>> Les Mikesell
>>>> lesmikesell@gmail.com
>>>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
>> For additional commands, e-mail: users-help@subversion.tigris.org
>>
>>
>>
>> This email is subject to our email legal notice, to view browse to
> http://www.angloplatinum.co.za/E-mailLegalNotice/
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
>> For additional commands, e-mail: users-help@subversion.tigris.org
>>
>>
>
>
> This email is subject to our email legal notice, to view browse to http://www.angloplatinum.co.za/E-mailLegalNotice/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
RE: RE: Re: Faster SVN Status
Posted by Niel De Clerk <ND...@Angloplat.com>.
Agreed - but that does explain why the first is faster than the second,
even though you don't get more information from the second - I can't
think of a reason why you'd want to use the second option rather than
the first, except if you're interested in only about 10% of the files in
the directory.
On a different note, is there a way to include subdirectories when using
wild-cards? I.e. use something like *.kb to include all files in the
current directory and all of its subdirectories?
Cheers
Niel
-----Original Message-----
From: Andy Levy [mailto:andy.levy@gmail.com]
Sent: 07 January 2008 13:55
To: Niel De Clerk
Cc: users@subversion.tigris.org
Subject: Re: RE: Re: Faster SVN Status
On Jan 7, 2008 12:21 AM, Niel De Clerk <ND...@angloplat.com> wrote:
> I definitely also saw the additional delay introduced with multiple
> transactions. Below are 2 ways of getting exactly the same thing.
> However the first give all the results followed by a single "Status
> against revision X" message where the second will give the "Status
> against revision X" for each individual file. To me this indicates
some
> additional handshaking? When on the same network as the repository,
> this is a non issue, but when on a remote connection (e.g. using VPN
> from home) this can make the difference between a 30 second and a 190
> second transaction.
>
> Svn status --show-updates --verbose -N c:\my-files
This asks for the status of the directory. One transaction.
> Svn status --show-updates --verbose c:\my-files\*.*
This asks for the status of each child of the directory because the
shell expands the wildcards. It's the same as if you were to run:
svn status --show-updates --verbose c:\my-files\file1
c:\my-files\file2 c:\my-files\file3 c:\my-files\directory1
It's not (IMHO) "exactly the same thing." You're asking for 2
different kinds of information.
>
> -----Original Message-----
> From: news [mailto:news@ger.gmane.org] On Behalf Of Vincent Tondellier
> Sent: 06 January 2008 01:51
> To: users@subversion.tigris.org
> Subject: RE: Re: Faster SVN Status
>
> Harvey, Edward wrote:
>
> > I can't say exactly what's happening in Niel's situation, but I
> support
> > a company with worldwide network, and when we SSH from Boston to
India
> > or vice-versa, there is a few second delay. Because there's a 250ms
> > ping response delay, and the initial ssh handshake is the equivalent
> of
> > 10-15 small packet round-trips, it's about 5-6 seconds before the
> > command prompt appears.
> >
> > If you use svn:// protocol and svnserve, there's just the basic tcp
> > handshake before useful data starts going across, so it's much
faster.
> > Call it 1-2 x the ping delay. About a half a second for me.
> >
> > If you use https://, it looks like something like half a dozen small
> > packet round trips. In my case, about 2 seconds delay. So it's
> faster
> > than svn+ssh, but not as fast as svnserve.
> >
>
> You should try the connection caching feature of OpenSSH, it only
> establish
> one master ssh connection, and opens new ssh "channels" for each new
> connection you make. The GCC wiki have some explanations on this :
>
> http://gcc.gnu.org/wiki/SSH connection caching
>
> >> -----Original Message-----
> >> From: Les Mikesell [mailto:lesmikesell@gmail.com]
> >> Sent: Friday, January 04, 2008 10:42 AM
> >> To: Steven Bakke
> >> Cc: Niel De Clerk; users@subversion.tigris.org
> >> Subject: Re: Faster SVN Status
> >>
> >> Steven Bakke wrote:
> >> >
> >> > However through profiling we found a performance bottleneck in
that
> >> it
> >> > starts a new repository session (network connection) once for
each
> >> > target of the command. In our case the problem was with 'svn
> >> update',
> >> > but I think it applies to almost anything. The performance
became
> >> > dominated by ~2-3 seconds per target overhead each time it made a
> > new
> >> > svn+ssh connection to the server.
> >>
> >> What can possibly cause a several-second delay? Does your reverse
> DNS
> >> lookup resolve quickly? Are IDENT queries dropped and timing out?
> >>
> >> --
> >> Les Mikesell
> >> lesmikesell@gmail.com
> >>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
>
>
>
> This email is subject to our email legal notice, to view browse to
http://www.angloplatinum.co.za/E-mailLegalNotice/
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
>
>
This email is subject to our email legal notice, to view browse to http://www.angloplatinum.co.za/E-mailLegalNotice/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: RE: Re: Faster SVN Status
Posted by Andy Levy <an...@gmail.com>.
On Jan 7, 2008 12:21 AM, Niel De Clerk <ND...@angloplat.com> wrote:
> I definitely also saw the additional delay introduced with multiple
> transactions. Below are 2 ways of getting exactly the same thing.
> However the first give all the results followed by a single "Status
> against revision X" message where the second will give the "Status
> against revision X" for each individual file. To me this indicates some
> additional handshaking? When on the same network as the repository,
> this is a non issue, but when on a remote connection (e.g. using VPN
> from home) this can make the difference between a 30 second and a 190
> second transaction.
>
> Svn status --show-updates --verbose -N c:\my-files
This asks for the status of the directory. One transaction.
> Svn status --show-updates --verbose c:\my-files\*.*
This asks for the status of each child of the directory because the
shell expands the wildcards. It's the same as if you were to run:
svn status --show-updates --verbose c:\my-files\file1
c:\my-files\file2 c:\my-files\file3 c:\my-files\directory1
It's not (IMHO) "exactly the same thing." You're asking for 2
different kinds of information.
>
> -----Original Message-----
> From: news [mailto:news@ger.gmane.org] On Behalf Of Vincent Tondellier
> Sent: 06 January 2008 01:51
> To: users@subversion.tigris.org
> Subject: RE: Re: Faster SVN Status
>
> Harvey, Edward wrote:
>
> > I can't say exactly what's happening in Niel's situation, but I
> support
> > a company with worldwide network, and when we SSH from Boston to India
> > or vice-versa, there is a few second delay. Because there's a 250ms
> > ping response delay, and the initial ssh handshake is the equivalent
> of
> > 10-15 small packet round-trips, it's about 5-6 seconds before the
> > command prompt appears.
> >
> > If you use svn:// protocol and svnserve, there's just the basic tcp
> > handshake before useful data starts going across, so it's much faster.
> > Call it 1-2 x the ping delay. About a half a second for me.
> >
> > If you use https://, it looks like something like half a dozen small
> > packet round trips. In my case, about 2 seconds delay. So it's
> faster
> > than svn+ssh, but not as fast as svnserve.
> >
>
> You should try the connection caching feature of OpenSSH, it only
> establish
> one master ssh connection, and opens new ssh "channels" for each new
> connection you make. The GCC wiki have some explanations on this :
>
> http://gcc.gnu.org/wiki/SSH connection caching
>
> >> -----Original Message-----
> >> From: Les Mikesell [mailto:lesmikesell@gmail.com]
> >> Sent: Friday, January 04, 2008 10:42 AM
> >> To: Steven Bakke
> >> Cc: Niel De Clerk; users@subversion.tigris.org
> >> Subject: Re: Faster SVN Status
> >>
> >> Steven Bakke wrote:
> >> >
> >> > However through profiling we found a performance bottleneck in that
> >> it
> >> > starts a new repository session (network connection) once for each
> >> > target of the command. In our case the problem was with 'svn
> >> update',
> >> > but I think it applies to almost anything. The performance became
> >> > dominated by ~2-3 seconds per target overhead each time it made a
> > new
> >> > svn+ssh connection to the server.
> >>
> >> What can possibly cause a several-second delay? Does your reverse
> DNS
> >> lookup resolve quickly? Are IDENT queries dropped and timing out?
> >>
> >> --
> >> Les Mikesell
> >> lesmikesell@gmail.com
> >>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
>
>
>
> This email is subject to our email legal notice, to view browse to http://www.angloplatinum.co.za/E-mailLegalNotice/
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Faster SVN Status
Posted by John Peacock <jo...@havurah-software.org>.
Niel De Clerk wrote:
> Thanks, I'll go for option 1.
> Relocating the repository is a simple as making a hot copy to the new
> location and pointing the service to the new location, right?
You don't need to hotcopy if you know that no transactions are taking
place (see next answer). And ordinary recursive file copy is fine. If
you were anywhere but Windows, I would recommend using rsync (since you
could rsync once while the repo was still active then a second time
after the service was disabled). You may find a native rsync for
Windows (but you probably shouldn't use Cygwin, since that can introduce
odd file permission issues)
> Is there an easy way to ensure no transactions take place during such a
> transfer?
Shut down the service on the public machine; no one is going to die if
the repo is offline for the time it takes to copy the files.
> Will locks etc remain in place, or should all clients first
> commit their last changes and release their locks?
If you copy the entire repository directory structure (or better yet,
zip it up and copy for back up purpose), you should notice nothing
except a speed improvement.
NOTE: I have never personally done this operation; I always install the
repo local to the service.
John
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
RE: Faster SVN Status
Posted by Niel De Clerk <ND...@Angloplat.com>.
Thanks, I'll go for option 1.
Relocating the repository is a simple as making a hot copy to the new
location and pointing the service to the new location, right?
Is there an easy way to ensure no transactions take place during such a
transfer? Will locks etc remain in place, or should all clients first
commit their last changes and release their locks?
Regards
Niel
-----Original Message-----
From: John Peacock [mailto:john.peacock@havurah-software.org]
Sent: 07 January 2008 14:26
To: Niel De Clerk
Cc: users@subversion.tigris.org
Subject: Re: Faster SVN Status
Niel De Clerk wrote:
> Our repository is on one machine and svnserve runs (as a windows
> service) on another machine (same network) using a mapped network
drive
> to the repository. Svnserve must run on the second machine because it
> has a fixed IP and is accessible through the firewalls to our various
> clients. The repository was installed on the other machine mainly
> because of backup etc reasons, but in theory it could be moved to the
> machine running svnserve (would this introduce a significant
performance
> gain?).
This is a non-optimal configuration and I am highly confident you would
see a
performance increase from moving the repository to the same server as
the
service itself. While your configuration will "work" (for some value of
work),
it is not efficient and there is even the possibility of corruption
(this is
true of any networked filesystem access like this).
You have two choices:
1) move the repository to be local to the svnserve process;
2) switch to using http[s] access, and run a proxy service on the
externally
accessible host and the Apache service on the box with the repo.
The second choice would keep your backup strategy intact, but would
require all
WC to be updated with the new path (you can use `svn switch --relocate`,
since
it is still the same repo). You also need to make sure that whatever
proxy you
install on the outwards facing box doesn't mess with the traffic (which
some/many do). See this link for more details:
http://subversion.tigris.org/faq.html#proxy
HTH
John
This email is subject to our email legal notice, to view browse to http://www.angloplatinum.co.za/E-mailLegalNotice/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Faster SVN Status
Posted by John Peacock <jo...@havurah-software.org>.
Niel De Clerk wrote:
> Our repository is on one machine and svnserve runs (as a windows
> service) on another machine (same network) using a mapped network drive
> to the repository. Svnserve must run on the second machine because it
> has a fixed IP and is accessible through the firewalls to our various
> clients. The repository was installed on the other machine mainly
> because of backup etc reasons, but in theory it could be moved to the
> machine running svnserve (would this introduce a significant performance
> gain?).
This is a non-optimal configuration and I am highly confident you would see a
performance increase from moving the repository to the same server as the
service itself. While your configuration will "work" (for some value of work),
it is not efficient and there is even the possibility of corruption (this is
true of any networked filesystem access like this).
You have two choices:
1) move the repository to be local to the svnserve process;
2) switch to using http[s] access, and run a proxy service on the externally
accessible host and the Apache service on the box with the repo.
The second choice would keep your backup strategy intact, but would require all
WC to be updated with the new path (you can use `svn switch --relocate`, since
it is still the same repo). You also need to make sure that whatever proxy you
install on the outwards facing box doesn't mess with the traffic (which
some/many do). See this link for more details:
http://subversion.tigris.org/faq.html#proxy
HTH
John
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
RE: RE: Re: Faster SVN Status
Posted by Niel De Clerk <ND...@Angloplat.com>.
Hi
Thanks for the replies.
We're using svnserve (which based on some replies I got is a good
thing).
Our repository is on one machine and svnserve runs (as a windows
service) on another machine (same network) using a mapped network drive
to the repository. Svnserve must run on the second machine because it
has a fixed IP and is accessible through the firewalls to our various
clients. The repository was installed on the other machine mainly
because of backup etc reasons, but in theory it could be moved to the
machine running svnserve (would this introduce a significant performance
gain?). We're mainly using IP addresses for referencing so DNS is not a
real issue.
I definitely also saw the additional delay introduced with multiple
transactions. Below are 2 ways of getting exactly the same thing.
However the first give all the results followed by a single "Status
against revision X" message where the second will give the "Status
against revision X" for each individual file. To me this indicates some
additional handshaking? When on the same network as the repository,
this is a non issue, but when on a remote connection (e.g. using VPN
from home) this can make the difference between a 30 second and a 190
second transaction.
Svn status --show-updates --verbose -N c:\my-files
Svn status --show-updates --verbose c:\my-files\*.*
Regards
Niel
-----Original Message-----
From: news [mailto:news@ger.gmane.org] On Behalf Of Vincent Tondellier
Sent: 06 January 2008 01:51
To: users@subversion.tigris.org
Subject: RE: Re: Faster SVN Status
Harvey, Edward wrote:
> I can't say exactly what's happening in Niel's situation, but I
support
> a company with worldwide network, and when we SSH from Boston to India
> or vice-versa, there is a few second delay. Because there's a 250ms
> ping response delay, and the initial ssh handshake is the equivalent
of
> 10-15 small packet round-trips, it's about 5-6 seconds before the
> command prompt appears.
>
> If you use svn:// protocol and svnserve, there's just the basic tcp
> handshake before useful data starts going across, so it's much faster.
> Call it 1-2 x the ping delay. About a half a second for me.
>
> If you use https://, it looks like something like half a dozen small
> packet round trips. In my case, about 2 seconds delay. So it's
faster
> than svn+ssh, but not as fast as svnserve.
>
You should try the connection caching feature of OpenSSH, it only
establish
one master ssh connection, and opens new ssh "channels" for each new
connection you make. The GCC wiki have some explanations on this :
http://gcc.gnu.org/wiki/SSH connection caching
>> -----Original Message-----
>> From: Les Mikesell [mailto:lesmikesell@gmail.com]
>> Sent: Friday, January 04, 2008 10:42 AM
>> To: Steven Bakke
>> Cc: Niel De Clerk; users@subversion.tigris.org
>> Subject: Re: Faster SVN Status
>>
>> Steven Bakke wrote:
>> >
>> > However through profiling we found a performance bottleneck in that
>> it
>> > starts a new repository session (network connection) once for each
>> > target of the command. In our case the problem was with 'svn
>> update',
>> > but I think it applies to almost anything. The performance became
>> > dominated by ~2-3 seconds per target overhead each time it made a
> new
>> > svn+ssh connection to the server.
>>
>> What can possibly cause a several-second delay? Does your reverse
DNS
>> lookup resolve quickly? Are IDENT queries dropped and timing out?
>>
>> --
>> Les Mikesell
>> lesmikesell@gmail.com
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
This email is subject to our email legal notice, to view browse to http://www.angloplatinum.co.za/E-mailLegalNotice/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
RE: Re: Faster SVN Status
Posted by Vincent Tondellier <to...@team1664.org>.
Harvey, Edward wrote:
> I can't say exactly what's happening in Niel's situation, but I support
> a company with worldwide network, and when we SSH from Boston to India
> or vice-versa, there is a few second delay. Because there's a 250ms
> ping response delay, and the initial ssh handshake is the equivalent of
> 10-15 small packet round-trips, it's about 5-6 seconds before the
> command prompt appears.
>
> If you use svn:// protocol and svnserve, there's just the basic tcp
> handshake before useful data starts going across, so it's much faster.
> Call it 1-2 x the ping delay. About a half a second for me.
>
> If you use https://, it looks like something like half a dozen small
> packet round trips. In my case, about 2 seconds delay. So it's faster
> than svn+ssh, but not as fast as svnserve.
>
You should try the connection caching feature of OpenSSH, it only establish
one master ssh connection, and opens new ssh "channels" for each new
connection you make. The GCC wiki have some explanations on this :
http://gcc.gnu.org/wiki/SSH connection caching
>> -----Original Message-----
>> From: Les Mikesell [mailto:lesmikesell@gmail.com]
>> Sent: Friday, January 04, 2008 10:42 AM
>> To: Steven Bakke
>> Cc: Niel De Clerk; users@subversion.tigris.org
>> Subject: Re: Faster SVN Status
>>
>> Steven Bakke wrote:
>> >
>> > However through profiling we found a performance bottleneck in that
>> it
>> > starts a new repository session (network connection) once for each
>> > target of the command. In our case the problem was with 'svn
>> update',
>> > but I think it applies to almost anything. The performance became
>> > dominated by ~2-3 seconds per target overhead each time it made a
> new
>> > svn+ssh connection to the server.
>>
>> What can possibly cause a several-second delay? Does your reverse DNS
>> lookup resolve quickly? Are IDENT queries dropped and timing out?
>>
>> --
>> Les Mikesell
>> lesmikesell@gmail.com
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
RE: Re: Faster SVN Status
Posted by "Harvey, Edward" <Ed...@patni.com>.
I can't say exactly what's happening in Niel's situation, but I support
a company with worldwide network, and when we SSH from Boston to India
or vice-versa, there is a few second delay. Because there's a 250ms
ping response delay, and the initial ssh handshake is the equivalent of
10-15 small packet round-trips, it's about 5-6 seconds before the
command prompt appears.
If you use svn:// protocol and svnserve, there's just the basic tcp
handshake before useful data starts going across, so it's much faster.
Call it 1-2 x the ping delay. About a half a second for me.
If you use https://, it looks like something like half a dozen small
packet round trips. In my case, about 2 seconds delay. So it's faster
than svn+ssh, but not as fast as svnserve.
> -----Original Message-----
> From: Les Mikesell [mailto:lesmikesell@gmail.com]
> Sent: Friday, January 04, 2008 10:42 AM
> To: Steven Bakke
> Cc: Niel De Clerk; users@subversion.tigris.org
> Subject: Re: Faster SVN Status
>
> Steven Bakke wrote:
> >
> > However through profiling we found a performance bottleneck in that
> it
> > starts a new repository session (network connection) once for each
> > target of the command. In our case the problem was with 'svn
> update',
> > but I think it applies to almost anything. The performance became
> > dominated by ~2-3 seconds per target overhead each time it made a
new
> > svn+ssh connection to the server.
>
> What can possibly cause a several-second delay? Does your reverse DNS
> lookup resolve quickly? Are IDENT queries dropped and timing out?
>
> --
> Les Mikesell
> lesmikesell@gmail.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
http://www.patni.com
World-Wide Partnerships. World-Class Solutions.
_____________________________________________________________________
This e-mail message may contain proprietary, confidential or legally
privileged information for the sole use of the person or entity to
whom this message was originally addressed. Any review, e-transmission
dissemination or other use of or taking of any action in reliance upon
this information by persons or entities other than the intended
recipient is prohibited. If you have received this e-mail in error
kindly delete this e-mail from your records. If it appears that this
mail has been forwarded to you without proper authority, please notify
us immediately at netadmin@patni.com and delete this mail.
_____________________________________________________________________
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Faster SVN Status
Posted by Les Mikesell <le...@gmail.com>.
Steven Bakke wrote:
>
> However through profiling we found a performance bottleneck in that it
> starts a new repository session (network connection) once for each
> target of the command. In our case the problem was with 'svn update',
> but I think it applies to almost anything. The performance became
> dominated by ~2-3 seconds per target overhead each time it made a new
> svn+ssh connection to the server.
What can possibly cause a several-second delay? Does your reverse DNS
lookup resolve quickly? Are IDENT queries dropped and timing out?
--
Les Mikesell
lesmikesell@gmail.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Faster SVN Status
Posted by Steven Bakke <st...@amd.com>.
On Jan 4, 2008, at 12:09 AM, Niel De Clerk wrote:
> Good day
> We use SVN over a large distributed network that’s normally slow and
> often unreliable. It has changed our lives drastically though –
> excellent product!
>
> The files that we store in the repository have various types and
> most of them remain unchanged (but it is nice having them there for
> initial deployment to new clients).
> The files in the repository that change regularly all have a .kb
> extension, and the applications running on our clients are mostly
> concerned with these files.
>
> Is it possible to do a “svn status –verbose –show-updates“ that will
> only list the *.kb files (including the ones in sub-directories)?
> I.e. is there a way (direct or workaround) to include a --targets
> switch for a status command? Or a way to ignore files (versioned or
> unversioned) when a certain filter/mask is provided?
What is your repository access method? We tend to need to do
something similar in our usage.
You can do the following:
% svn stat -u -v path1/*.kb path2/*.kb ...
However through profiling we found a performance bottleneck in that it
starts a new repository session (network connection) once for each
target of the command. In our case the problem was with 'svn
update', but I think it applies to almost anything. The performance
became dominated by ~2-3 seconds per target overhead each time it made
a new svn+ssh connection to the server. You can imagine that if this
is done on several thousand files, that it becomes a real drag.
On the other hand, it may be faster to just give it targeted
directories, since it will make one connection for that directory
though it will report status for the entire contents. You can add -N
to prevent this from being recursive too.
> Will this improve the speed of the command execution? (Note this
> will reduce the number of files to consider by more than half)
> I’ve tried to the following syntax c:\myfiles\*.kb, but this seems
> to do a separate transaction for each file and is much slower.
>
> Many thanks
> Niel
>
> This email is subject to our email legal notice, to view click here