You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Magnus Torfason <zu...@gmail.com> on 2010/01/11 16:53:55 UTC
svn+ssh issues with svn 1.6.6 on windows (especially for svn-external
repo)
I wanted to send a word to the developers about strange behavior by
Subversion on Windows when using svn+ssh to connect to an external
(linux) host, with repositories that contain a number of svn-externals
entries.
I recently upgraded both SlikSVN and TortoiseSVN from 1.6.1 to 1.6.6.
After the upgrade I found intermittent problems with running svn update
(TSVN), and a persistent problem with running svnsync (SSVN).
I will go through each of them in turn below, because it seems to me
that they might be related (although I'm not sure).
1: svnsync (through SSVN):
A svnsync process from an external server that had always worked did no
longer complete successfully. The error message was:
plink: unknown option "-q"
Connection terminated
I don't know whether the plink error came from plink or if it is a
pass-through from some other process, but looking at the plink
documentation, I can't see that there has ever been a "-q" option.
Resolution: I downgraded SlikSVN to 1.6.5, which did not resolve the
issue. However, after downgrading all the way down to 1.6.1 again, this
issue seems to have been resolved.
2: svn update (through TSVN):
At the same time, I started having intermittent "connection refused"
problems when updating my repository from the same server. This always
happened while updating one of the several svn-external directories that
I have as part of my repository. After the error, I could not ssh into
the server either, until after a few minutes. Downgrading to 1.6.1 did
not resolve this problem.
Resolution: None that is entirely satisfying.
Workaround: Choose "Omit externals" when updating.
Further thoughts: After reading the thread on Issue #2580 I started
wondering if the issue might be caused by a quota on open/hanging ssh
connections that was enforced by the server (I'm not an admin on the
server so I can't do much to look into that). But if the following is
the case:
> Instantly killing the tunnel (clearly the WRONG way to
> terminate a TCP connection!) sets off a chain of events, ALL
> OF WHICH ARE ERRORS. The svn protocol does not complete
> normally, nor does the ssh protocol, nor does the TCP
> protocol! Do we want EVERY SINGLE svn+ssh connection to
> complete through error paths? That's what's happening now,
> to EVERYONE who is using svn+ssh from a Windows client! I've
> already explained why we haven't gotten more complaints.
>
> But I repeat: instantly killing the tunnel upon receipt of
> the last SVN data from a remote svnserve is NOT the way to
> shut down the connection between the two. It leaves the
> remote sshd wondering what happened and sitting there
> waiting for the next step in ITS protocol (SSH). It has no
> idea what is coming next (more data or what???). So it sits
> there waiting...
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2410204
Then it seems reasonable that defensive measures by server
administrators might lead to this problem in the case of a number of ssh
connections being rapidly opened and then closed in an unclean manner
(since svn-update seems to open a new connection for each svn-external
entry).
Best,
Magnus
RE: svn+ssh issues on Windows due to passing -q by default
Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: Magnus Torfason [mailto:zulutime.net@gmail.com]
> Sent: donderdag 11 februari 2010 19:19
> To: dev@subversion.apache.org
> Cc: Daniel Shahaf; rhuijben@open.collab.net
> Subject: svn+ssh issues on Windows due to passing -q by default
>
> Hi, I sent an email a while ago detailing problems with connecting from
> a Windows client to a *nix server over ssh:
<snip>
> fixes this issue for me, but it means that TortoiseSVN also uses the
> SlikSvn plink, which is not really what I want. It seems to me that the
> best solution would be to change the behavior in the SlikSvn
> distribution for Windows, either by removing the -q from the default
> settings, or by patching the plink (named ssh.exe and presumably
> custom-built) that ships with SlikSvn.exe so that it will ignore the -q
> switch rather than throw an error.
SlikSVN doesn't distribute a ssh.exe. You probably got that exe from some
other place.
(I build the SlikSVN MSIs and they don't contain a ssh.exe)
SharpSvn has its own plink, but that one is not called ssh.exe. And since
the last few releases that version ignores -q.
Bert
Re: svn+ssh issues on Windows due to passing -q by default
Posted by Magnus Torfason <zu...@gmail.com>.
Hi all,
I just wanted to note that I stand corrected on the issue below. It
turns out that SlikSvn does not distribute an ssh.exe. I must have put
it there manually and then forgot all about it. So the message really
should have gone to the user list, not the dev list.
Sorry about the noise.
Best,
Magnus
On 2/11/2010 1:19 PM, Magnus Torfason wrote:
> Hi, I sent an email a while ago detailing problems with connecting from
> a Windows client to a *nix server over ssh:
>
> On 1/11/2010 5:20 PM, Daniel Shahaf wrote:
>> Magnus Torfason wrote on Mon, 11 Jan 2010 at 11:53 -0500:
>>> 1: svnsync (through SSVN):
>>>
>>> A svnsync process from an external server that had always worked did no
>>> longer complete successfully. The error message was:
>>>
>>> plink: unknown option "-q"
>>> Connection terminated
>>>
>>
>> We added the '-q' to the default value of the 'ssh' svn+*:// scheme some
>> time ago. The help now reads:
>>
>> ...
>>
>> Related?
>>
>> Daniel
>> (my configuration overrides it as follows:
>> ssh = $SVN_SSH /path/to/plink.exe -batch -ssh -2
>> )
>>
>
> Yes, related.
>
> After more testing it seems clear that this has to do with SlikSvn
> distributing an ssh.exe that does not accept the -q switch, but with the
> default setting being to pass the -q.
>
> TortoiseSVN ships with its own version of plink, and does not suffer
> from this issue (but copying that version over does not seem to fix it
> for me).
>
> Changing the config file to read
> ssh = $SVN_SSH ssh
> fixes this issue for me, but it means that TortoiseSVN also uses the
> SlikSvn plink, which is not really what I want. It seems to me that the
> best solution would be to change the behavior in the SlikSvn
> distribution for Windows, either by removing the -q from the default
> settings, or by patching the plink (named ssh.exe and presumably
> custom-built) that ships with SlikSvn.exe so that it will ignore the -q
> switch rather than throw an error.
>
> Best,
> Magnus
svn+ssh issues on Windows due to passing -q by default
Posted by Magnus Torfason <zu...@gmail.com>.
Hi, I sent an email a while ago detailing problems with connecting from
a Windows client to a *nix server over ssh:
On 1/11/2010 5:20 PM, Daniel Shahaf wrote:
> Magnus Torfason wrote on Mon, 11 Jan 2010 at 11:53 -0500:
>> 1: svnsync (through SSVN):
>>
>> A svnsync process from an external server that had always worked did no
>> longer complete successfully. The error message was:
>>
>> plink: unknown option "-q"
>> Connection terminated
>>
>
> We added the '-q' to the default value of the 'ssh' svn+*:// scheme some
> time ago. The help now reads:
>
> ...
>
> Related?
>
> Daniel
> (my configuration overrides it as follows:
> ssh = $SVN_SSH /path/to/plink.exe -batch -ssh -2
> )
>
Yes, related.
After more testing it seems clear that this has to do with SlikSvn
distributing an ssh.exe that does not accept the -q switch, but with the
default setting being to pass the -q.
TortoiseSVN ships with its own version of plink, and does not suffer
from this issue (but copying that version over does not seem to fix it
for me).
Changing the config file to read
ssh = $SVN_SSH ssh
fixes this issue for me, but it means that TortoiseSVN also uses the
SlikSvn plink, which is not really what I want. It seems to me that the
best solution would be to change the behavior in the SlikSvn
distribution for Windows, either by removing the -q from the default
settings, or by patching the plink (named ssh.exe and presumably
custom-built) that ships with SlikSvn.exe so that it will ignore the -q
switch rather than throw an error.
Best,
Magnus
Re: svn+ssh issues with svn 1.6.6 on windows (especially for
svn-external repo)
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Magnus Torfason wrote on Mon, 11 Jan 2010 at 11:53 -0500:
> 1: svnsync (through SSVN):
>
> A svnsync process from an external server that had always worked did no
> longer complete successfully. The error message was:
>
> plink: unknown option "-q"
> Connection terminated
>
> I don't know whether the plink error came from plink or if it is a
> pass-through from some other process, but looking at the plink
> documentation, I can't see that there has ever been a "-q" option.
>
> Resolution: I downgraded SlikSVN to 1.6.5, which did not resolve the issue.
> However, after downgrading all the way down to 1.6.1 again, this issue seems
> to have been resolved.
We added the '-q' to the default value of the 'ssh' svn+*:// scheme some
time ago. The help now reads:
"### Section for configuring tunnel agents." NL
"[tunnels]" NL
"### Configure svn protocol tunnel schemes here. By default, only" NL
"### the 'ssh' scheme is defined. You can define other schemes to" NL
"### be used with 'svn+scheme://hostname/path' URLs. A scheme" NL
"### definition is simply a command, optionally prefixed by an" NL
"### environment variable name which can override the command if it" NL
"### is defined. The command (or environment variable) may contain" NL
"### arguments, using standard shell quoting for arguments with" NL
"### spaces. The command will be invoked as:" NL
"### <command> <hostname> svnserve -t" NL
"### (If the URL includes a username, then the hostname will be" NL
"### passed to the tunnel agent as <user>@<hostname>.) If the" NL
"### built-in ssh scheme were not predefined, it could be defined" NL
"### as:" NL
"# ssh = $SVN_SSH ssh -q" NL
Related?
Daniel
(my configuration overrides it as follows:
ssh = $SVN_SSH /path/to/plink.exe -batch -ssh -2
)
svn+ssh issues when updating repo with multiple externals
Posted by Magnus Torfason <zu...@gmail.com>.
Hi, I just wanted to give an update on the issue I reported below. I've
tried the same update with 1.6.9, and I am still having the same problem.
Furthermore, when I use an alternative ssh client with TortoiseSVN (the
alternative pops up a command line when run), I can clearly observe a
separate ssh instance being launched for each external in my repo.
I am continuing to use "omit externals", and thinking about doing
something on the server side to get around this issue, but I felt an
update was warranted nevertheless, since it really does feel like the
client is overusing resources here.
Best,
Magnus
On 1/11/2010 2:57 PM, kmradke@rockwellcollins.com wrote:
>
> Magnus Torfason <zu...@gmail.com> wrote on 01/11/2010 01:39:18 PM:
>
> > After some further exploration, I have an update on my issue with svn up
> > failing on my svn-externals-infested repository. Below, I report an
> > alternative workaround, and the output of ps while doing the update.
> >
> > On 1/11/2010 11:53 AM, Magnus Torfason wrote:
> > >
> > > 2: svn update (through TSVN):
> > >
> > > At the same time, I started having intermittent "connection refused"
> > > problems when updating my repository from the same server. This always
> > > happened while updating one of the several svn-external directories
> > > that
> > > I have as part of my repository. After the error, I could not ssh into
> > > the server either, until after a few minutes. Downgrading to 1.6.1 did
> > > not resolve this problem.
> > >
> > > Resolution: None that is entirely satisfying.
> > > Workaround: Choose "Omit externals" when updating.
> >
> > Alternative Workaround: If I close pageant, I get prompted on each new
> > ssh connection to the server. By entering my password only after a ten
> > second delay for each prompt, the process completes successfully.
> > Entering the password quickly leads to the same behavior as if pageant
> > is running.
>
> You may be running into the problem I saw (and fixed with the hopefully
> soon to be released 1.6.8) where each svn:external in an update would
> open and not correctly close the network connection until the update
> finished. This would cause Windows XP to disallow creation of new
> network connections because of the large number of open connections
> on the client. We had thousands of externals and when you would try
> and do an update you would get weird "can not connect to server" errors.
>
> Kevin R.
Re: svn+ssh issues with svn 1.6.6 on windows (especially for svn-external
repo)
Posted by km...@rockwellcollins.com.
Magnus Torfason <zu...@gmail.com> wrote on 01/11/2010 01:39:18 PM:
> After some further exploration, I have an update on my issue with svn up
> failing on my svn-externals-infested repository. Below, I report an
> alternative workaround, and the output of ps while doing the update.
>
> On 1/11/2010 11:53 AM, Magnus Torfason wrote:
> >
> > 2: svn update (through TSVN):
> >
> > At the same time, I started having intermittent "connection refused"
> > problems when updating my repository from the same server. This always
> > happened while updating one of the several svn-external directories
that
> > I have as part of my repository. After the error, I could not ssh into
> > the server either, until after a few minutes. Downgrading to 1.6.1 did
> > not resolve this problem.
> >
> > Resolution: None that is entirely satisfying.
> > Workaround: Choose "Omit externals" when updating.
>
> Alternative Workaround: If I close pageant, I get prompted on each new
> ssh connection to the server. By entering my password only after a ten
> second delay for each prompt, the process completes successfully.
> Entering the password quickly leads to the same behavior as if pageant
> is running.
You may be running into the problem I saw (and fixed with the hopefully
soon to be released 1.6.8) where each svn:external in an update would
open and not correctly close the network connection until the update
finished. This would cause Windows XP to disallow creation of new
network connections because of the large number of open connections
on the client. We had thousands of externals and when you would try
and do an update you would get weird "can not connect to server" errors.
Kevin R.
Re: svn+ssh issues with svn 1.6.6 on windows (especially for svn-external
repo)
Posted by Magnus Torfason <zu...@gmail.com>.
After some further exploration, I have an update on my issue with svn up
failing on my svn-externals-infested repository. Below, I report an
alternative workaround, and the output of ps while doing the update.
On 1/11/2010 11:53 AM, Magnus Torfason wrote:
>
> 2: svn update (through TSVN):
>
> At the same time, I started having intermittent "connection refused"
> problems when updating my repository from the same server. This always
> happened while updating one of the several svn-external directories that
> I have as part of my repository. After the error, I could not ssh into
> the server either, until after a few minutes. Downgrading to 1.6.1 did
> not resolve this problem.
>
> Resolution: None that is entirely satisfying.
> Workaround: Choose "Omit externals" when updating.
Alternative Workaround: If I close pageant, I get prompted on each new
ssh connection to the server. By entering my password only after a ten
second delay for each prompt, the process completes successfully.
Entering the password quickly leads to the same behavior as if pageant
is running.
Output of ps: I also tried running ps -U <username> on the server while
the password prompts were being displayed, and I found that a new
lt-svnserve process is started for each svn-external, and they are not
closed until after the operation completes (the output of these commands
is included at the bottom of this email).
So it definitely seems to be an issue resulting from some sort of
throttling on the server. Of course, one could argue that such
throttling is unreasonable. However, it could also be argued that it is
unreasonable to open a new ssh connection for each external, without
closing the ones that are finished.
NOTE 1: These are *not* nested externals, all externals are "side by
side" in the main repository, and refer to the same (small) directory
within the same repo.
NOTE 2: I can open putty terminals to the same server willy nilly
without problems, so it is not simply an issue of a limit on the number
of ssh connections as such. It may be a limit on the number of ssh
connections in a particular state (hanging, etc.) or something else.
Best,
Magnus
###########################################
# Output of ps -U <username> while completing a svn up
# operation through TSVN
###########################################
# # First password prompt
# ps -U <username>
PID TTY TIME CMD
18444 ? 00:00:00 sshd
18472 pts/9 00:00:00 bash
18791 ? 00:00:00 sshd
18807 ? 00:00:00 lt-svnserve
20074 pts/9 00:00:00 ps
# # First external (second password prompt)
# ps -U <username>
PID TTY TIME CMD
18444 ? 00:00:00 sshd
18472 pts/9 00:00:00 bash
18791 ? 00:00:00 sshd
18807 ? 00:00:00 lt-svnserve
21397 ? 00:00:00 sshd
21407 ? 00:00:00 lt-svnserve
22095 pts/9 00:00:00 ps
# # Second external
# ps -U <username>
PID TTY TIME CMD
18444 ? 00:00:00 sshd
18472 pts/9 00:00:00 bash
18791 ? 00:00:00 sshd
18807 ? 00:00:00 lt-svnserve
21397 ? 00:00:00 sshd
21407 ? 00:00:00 lt-svnserve
24501 ? 00:00:00 sshd
24514 ? 00:00:00 lt-svnserve
24945 pts/9 00:00:00 ps
# # Third external
# ps -U <username>
PID TTY TIME CMD
18444 ? 00:00:00 sshd
18472 pts/9 00:00:00 bash
18791 ? 00:00:00 sshd
18807 ? 00:00:00 lt-svnserve
21397 ? 00:00:00 sshd
21407 ? 00:00:00 lt-svnserve
24501 ? 00:00:00 sshd
24514 ? 00:00:00 lt-svnserve
25991 ? 00:00:00 sshd
26005 ? 00:00:00 lt-svnserve
26261 pts/9 00:00:00 ps
# # Fourth external
# ps -U <username>
PID TTY TIME CMD
18444 ? 00:00:00 sshd
18472 pts/9 00:00:00 bash
18791 ? 00:00:00 sshd
18807 ? 00:00:00 lt-svnserve
21397 ? 00:00:00 sshd
21407 ? 00:00:00 lt-svnserve
24501 ? 00:00:00 sshd
24514 ? 00:00:00 lt-svnserve
25991 ? 00:00:00 sshd
26005 ? 00:00:00 lt-svnserve
28945 ? 00:00:00 sshd
28957 ? 00:00:00 lt-svnserve
29157 pts/9 00:00:00 ps
# # Fifth external
# ps -U <username>
PID TTY TIME CMD
18444 ? 00:00:00 sshd
18472 pts/9 00:00:00 bash
18791 ? 00:00:00 sshd
18807 ? 00:00:00 lt-svnserve
21397 ? 00:00:00 sshd
21407 ? 00:00:00 lt-svnserve
24501 ? 00:00:00 sshd
24514 ? 00:00:00 lt-svnserve
25991 ? 00:00:00 sshd
26005 ? 00:00:00 lt-svnserve
28945 ? 00:00:00 sshd
28957 ? 00:00:00 lt-svnserve
30962 ? 00:00:00 sshd
30975 ? 00:00:00 lt-svnserve
31222 pts/9 00:00:00 ps
# # Six instances of svnserve running !
#
# # ...
#
# # Subversion finished
# ps -U <username>
PID TTY TIME CMD
662 pts/9 00:00:00 ps
18444 ? 00:00:00 sshd
18472 pts/9 00:00:00 bash