svn update via HTTPS works 95% of the time, then randomly shanks, "issuer not trusted"

[Dan Yost <yo...@gmail.com>]

Greetings,

I'm at my wit's end trying to debug a troubling, highly irregular
issue connecting to a SVN server via HTTPS. I am very confident this
is a client issue, not a server issue. The server is CollabNet's
Subversion Edge server, which has 1.6.15 built in. The client is
1.6.15 running under Cygwin. I have many, many clients, most of which
are running Windows Server 2003 (if that matters)--all Cygwin, all
this same package. Lots of traffic every day. Almost all of it works
just fine.

Now, a very important fact: the svn operations WORK 95% of the time.
They just work. Everything is absolutely fine. This is an unattended,
fully-automated environment, launched from cron (well, Task Scheduler
when running on these Windows boxes). 95% of the time, the job
launches, runs my "svn update" command from within one of its working
copies:

svn update --username=some --password=thing --non-interactive


and it works beautifully.

Every "once in awhile," for absolutely no reason I can find (have
looked at logs, logs, logs, timing, the stars and planets,
everything), with absolutely nothing that I can find having changed,
it dies with this:


svn: OPTIONS of 'https://myserver:9999/svn/some/path': Server
certificate verification failed: certificate issued for a different
hostname, issuer is not trusted (https://myserver:9999)


(I have changed the paths/ports to protect the innocent).


Now for most of my clients, the "issued for a different hostname" part
is omitted--they still have the "issuer not trusted" part. That's
because my one host that I'm really digging in to debug this has to
use a different hostname for certain reasons. So, focus on the "issuer
not trusted" part because that's what we see across all clients.

But keep in mind: it works. It works 95% of the time. I change
absolutely nothing. Having found no indication of why it will
*randomly* fail with the above error, I decided to add the
"--trust-cerver-cert" flag, thinking surely this will eliminate all
woes. Just trust it and go (trust it like you already do 95% of the
time anyway)!


svn update --username=some --password=thing --non-interactive
--trust-server-cert


But, nope, I randomly get the very same error.

These key facts:

1) It works 95% of the time (well, maybe 85%)

2) I can find NOTHING changing (on client or server) between a working
op and a breaking op--it's all automated, and just randomly shanks


are leading me to believe that this error message is actually a
complete wild goose chase, and that the problem is not actually
cert-based but is something else (with a wrong error message). In
looking at the server HTTP logs, it does indeed appear that the client
bails out before continuing the connection--there's no log, meaning it
does appear that the client can't handshake (or refuses to), and then
quits.

I have thought about adding options to config (on the command line)
such as ssl-trust-default-ca, but again, I already told svn to
--trust-server-cert and it randomly still dies, so I don't think that
will help.

You'll be tempted to focus on SSL cert issues on the server or
something, but remember that we don't change anything and it works
fine almost all the time! (I'm still open to any thoughts of course).

Can anybody point me to some other debug options, some other flag, or
any other way to determine why, 15% of the time, it dies, yet 85% of
the time it updates just fine?

Thanks,
Dan

Re: svn update via HTTPS works 95% of the time, then randomly shanks, "issuer not trusted"

[Dan Yost <yo...@gmail.com>]

On Tue, Jul 26, 2011 at 2:19 PM, Daniel Shahaf <d....@daniel.shahaf.name> wrote:
> Dan Yost wrote on Tue, Jul 26, 2011 at 12:57:29 -0500:
>> Or to state the below (pardon the top-post) much more simply: the
>> --trust-server-cert flag does not work. It fails to perform its
>> singular function, which is...to force trust of the server cert,
>> right?
>>
>
> Its function is to accept certificates signed by unknown CA's without
> prompting.  In your case you have two failures, one of them being the
> mismatching subject name (hostname), so you do get prompted.
>


Ah, so indeed perhaps this will reveal a key difference between my
test client and the rest of the live clients (still gathering info on
them to see if they continue to fail). The rest of the clients don't
have the hostname issue, I believe, so perhaps they will indeed be
improved by the flag.

But, any idea why it works one minute, then doesn't the next, when
nobody touches anything? Or, a good way to dig deeper? Perhaps I need
a very verbose dump of the whole SSL handshake, I guess, to find out
why it works fine one minute then doesn't the next, but I didn't find
an option (particularly on the "update" subcommand).

Thanks,
Dan

Re: svn update via HTTPS works 95% of the time, then randomly shanks, "issuer not trusted"

[Geoff Hoffman <gh...@cardinalpath.com>]

On Tue, Jul 26, 2011 at 1:20 PM, Dan Yost <yo...@gmail.com> wrote:

> On Tue, Jul 26, 2011 at 3:11 PM, Geoff Hoffman
> <gh...@cardinalpath.com> wrote:
> > Long shot here... this is probably off base, as I am not that experienced
> > with lower-level SSL problems, but are you by chance using an issuer that
> > provides an intermediary certificate?
> > For example, to install an SSL cert from GoDaddy, you have to also
> include
> > the gd_bundle.crt. The Wikipedia article below makes me wonder if there
> is
> > just some network hiccups sometimes, trying to validate your certificate
> > chain authority.
> >>
> >> From http://en.wikipedia.org/wiki/Intermediate_certificate_authorities
> >> If the certificate was not issued by a trusted CA, the connecting device
> >> (e.g., a web browser) will then check to see if the issuing CA of the
> >> certificate was issued by a trusted CA, and so on until either a trusted
> CA
> >> is found (at which point a trusted, secure connection will be
> established)
> >> or no trusted CA can be found (at which point the device will usually
> >> display an error).
> >
> >
>
>
> Yes, and indeed this is a GoDaddy cert, with the bundle installed to
> keep the chain intact, so thus it does work that 95% of the time. I
> was thinking that the chain is all presented from the server to client
> in one fell swoop, with no need to go fetch anything else "out there"
> (not that you're suggesting that is what it needs to do--go outside to
> fetch something). But indeed, I suppose it could complicate the
> handshake in such a way as to cause this intermittent failure--would
> really like to be able to "watch" that happen via some kind of verbose
> log.
>
> Dan
>


If this *might* be the problem, I'm guessing that browsers do a better job
of "trying again a few times" than the svn client might. If you can simulate
what your workflow is doing in Firefox with the LiveHeaders plugin, you can
distill it down to a list of FQDNs that are required, then tracert them,
ping them, etc., to see if you have any dropped packets. Also IPs are a
straighter path than DNS names. It may not be easy for you to change
everything around, but if you switched it all to IP-based you could rule out
DNS being a problem.

Re: svn update via HTTPS works 95% of the time, then randomly shanks, "issuer not trusted"

[Dan Yost <yo...@gmail.com>]

On Tue, Jul 26, 2011 at 3:11 PM, Geoff Hoffman
<gh...@cardinalpath.com> wrote:
> Long shot here... this is probably off base, as I am not that experienced
> with lower-level SSL problems, but are you by chance using an issuer that
> provides an intermediary certificate?
> For example, to install an SSL cert from GoDaddy, you have to also include
> the gd_bundle.crt. The Wikipedia article below makes me wonder if there is
> just some network hiccups sometimes, trying to validate your certificate
> chain authority.
>>
>> From http://en.wikipedia.org/wiki/Intermediate_certificate_authorities
>> If the certificate was not issued by a trusted CA, the connecting device
>> (e.g., a web browser) will then check to see if the issuing CA of the
>> certificate was issued by a trusted CA, and so on until either a trusted CA
>> is found (at which point a trusted, secure connection will be established)
>> or no trusted CA can be found (at which point the device will usually
>> display an error).
>
>


Yes, and indeed this is a GoDaddy cert, with the bundle installed to
keep the chain intact, so thus it does work that 95% of the time. I
was thinking that the chain is all presented from the server to client
in one fell swoop, with no need to go fetch anything else "out there"
(not that you're suggesting that is what it needs to do--go outside to
fetch something). But indeed, I suppose it could complicate the
handshake in such a way as to cause this intermittent failure--would
really like to be able to "watch" that happen via some kind of verbose
log.

Dan

Re: svn update via HTTPS works 95% of the time, then randomly shanks, "issuer not trusted"

[Geoff Hoffman <gh...@cardinalpath.com>]

Long shot here... this is probably off base, as I am not that experienced
with lower-level SSL problems, but are you by chance using an issuer that
provides an intermediary certificate?

For example, to install an SSL cert from GoDaddy, you have to also include
the gd_bundle.crt. The Wikipedia article below makes me wonder if there is
just some network hiccups sometimes, trying to validate your certificate
chain authority.

>From http://en.wikipedia.org/wiki/Intermediate_certificate_authorities
> If the certificate was not issued by a trusted CA, the connecting device
> (e.g., a web browser) will then check to see if the issuing CA of the
> certificate was issued by a trusted CA, and so on until either a trusted CA
> is found (at which point a trusted, secure connection will be established)
> or no trusted CA can be found (at which point the device will usually
> display an error).

Re: svn update via HTTPS works 95% of the time, then randomly shanks, "issuer not trusted"

[Dan Yost <yo...@gmail.com>]

On Tue, Jul 26, 2011 at 2:32 PM, Daniel Shahaf <d....@daniel.shahaf.name> wrote:
> Daniel Shahaf wrote on Tue, Jul 26, 2011 at 22:19:13 +0300:
>> Dan Yost wrote on Tue, Jul 26, 2011 at 12:57:29 -0500:
>> > Or to state the below (pardon the top-post) much more simply: the
>> > --trust-server-cert flag does not work. It fails to perform its
>> > singular function, which is...to force trust of the server cert,
>> > right?
>> >
>>
>> Its function is to accept certificates signed by unknown CA's without
>> prompting.  In your case you have two failures, one of them being the
>> mismatching subject name (hostname), so you do get prompted.
>
> See ssl_trust_unknown_server_cert() in subversion/libsvn_subr/cmdline.c.
>
> There were discussions about extending this to, for example,
> --trust-server-cert2=comma,separated,list,of,failures,to,ignore ,
> and I think someone may have started working on a patch, but they never
> submitted it to us.
>


A possibility, indeed. What's still very odd is that there's actually
no error--nothing to ignore. That is, 95% of the time, then randomly
(5%) it shanks, and nobody touched anything (on either server side or
client side). That's what is making me crazy.

Dan

Re: svn update via HTTPS works 95% of the time, then randomly shanks, "issuer not trusted"

[Daniel Shahaf <d....@daniel.shahaf.name>]

Daniel Shahaf wrote on Tue, Jul 26, 2011 at 22:19:13 +0300:
> Dan Yost wrote on Tue, Jul 26, 2011 at 12:57:29 -0500:
> > Or to state the below (pardon the top-post) much more simply: the
> > --trust-server-cert flag does not work. It fails to perform its
> > singular function, which is...to force trust of the server cert,
> > right?
> > 
> 
> Its function is to accept certificates signed by unknown CA's without
> prompting.  In your case you have two failures, one of them being the
> mismatching subject name (hostname), so you do get prompted.

See ssl_trust_unknown_server_cert() in subversion/libsvn_subr/cmdline.c.

There were discussions about extending this to, for example,
--trust-server-cert2=comma,separated,list,of,failures,to,ignore ,
and I think someone may have started working on a patch, but they never
submitted it to us.

Re: svn update via HTTPS works 95% of the time, then randomly shanks, "issuer not trusted"

[Daniel Shahaf <d....@daniel.shahaf.name>]

Dan Yost wrote on Tue, Jul 26, 2011 at 12:57:29 -0500:
> Or to state the below (pardon the top-post) much more simply: the
> --trust-server-cert flag does not work. It fails to perform its
> singular function, which is...to force trust of the server cert,
> right?
> 

Its function is to accept certificates signed by unknown CA's without
prompting.  In your case you have two failures, one of them being the
mismatching subject name (hostname), so you do get prompted.

> (Though I still think something else is going on, I can start there).
> 
> 
> 
> On Mon, Jul 25, 2011 at 1:12 PM, Dan Yost <yo...@gmail.com> wrote:
> > Greetings,
> >
> > I'm at my wit's end trying to debug a troubling, highly irregular
> > issue connecting to a SVN server via HTTPS. I am very confident this
> > is a client issue, not a server issue. The server is CollabNet's
> > Subversion Edge server, which has 1.6.15 built in. The client is
> > 1.6.15 running under Cygwin. I have many, many clients, most of which
> > are running Windows Server 2003 (if that matters)--all Cygwin, all
> > this same package. Lots of traffic every day. Almost all of it works
> > just fine.
> >
> > Now, a very important fact: the svn operations WORK 95% of the time.
> > They just work. Everything is absolutely fine. This is an unattended,
> > fully-automated environment, launched from cron (well, Task Scheduler
> > when running on these Windows boxes). 95% of the time, the job
> > launches, runs my "svn update" command from within one of its working
> > copies:
> >
> > svn update --username=some --password=thing --non-interactive
> >
> >
> > and it works beautifully.
> >
> > Every "once in awhile," for absolutely no reason I can find (have
> > looked at logs, logs, logs, timing, the stars and planets,
> > everything), with absolutely nothing that I can find having changed,
> > it dies with this:
> >
> >
> > svn: OPTIONS of 'https://myserver:9999/svn/some/path': Server
> > certificate verification failed: certificate issued for a different
> > hostname, issuer is not trusted (https://myserver:9999)
> >
> >
> > (I have changed the paths/ports to protect the innocent).
> >
> >
> > Now for most of my clients, the "issued for a different hostname" part
> > is omitted--they still have the "issuer not trusted" part. That's
> > because my one host that I'm really digging in to debug this has to
> > use a different hostname for certain reasons. So, focus on the "issuer
> > not trusted" part because that's what we see across all clients.
> >
> > But keep in mind: it works. It works 95% of the time. I change
> > absolutely nothing. Having found no indication of why it will
> > *randomly* fail with the above error, I decided to add the
> > "--trust-cerver-cert" flag, thinking surely this will eliminate all
> > woes. Just trust it and go (trust it like you already do 95% of the
> > time anyway)!
> >
> >
> > svn update --username=some --password=thing --non-interactive
> > --trust-server-cert
> >
> >
> > But, nope, I randomly get the very same error.
> >
> > These key facts:
> >
> > 1) It works 95% of the time (well, maybe 85%)
> >
> > 2) I can find NOTHING changing (on client or server) between a working
> > op and a breaking op--it's all automated, and just randomly shanks
> >
> >
> > are leading me to believe that this error message is actually a
> > complete wild goose chase, and that the problem is not actually
> > cert-based but is something else (with a wrong error message). In
> > looking at the server HTTP logs, it does indeed appear that the client
> > bails out before continuing the connection--there's no log, meaning it
> > does appear that the client can't handshake (or refuses to), and then
> > quits.
> >
> > I have thought about adding options to config (on the command line)
> > such as ssl-trust-default-ca, but again, I already told svn to
> > --trust-server-cert and it randomly still dies, so I don't think that
> > will help.
> >
> > You'll be tempted to focus on SSL cert issues on the server or
> > something, but remember that we don't change anything and it works
> > fine almost all the time! (I'm still open to any thoughts of course).
> >
> > Can anybody point me to some other debug options, some other flag, or
> > any other way to determine why, 15% of the time, it dies, yet 85% of
> > the time it updates just fine?
> >
> > Thanks,
> > Dan
> >

Re: svn update via HTTPS works 95% of the time, then randomly shanks, "issuer not trusted"

[Dan Yost <yo...@gmail.com>]

Or to state the below (pardon the top-post) much more simply: the
--trust-server-cert flag does not work. It fails to perform its
singular function, which is...to force trust of the server cert,
right?

(Though I still think something else is going on, I can start there).



On Mon, Jul 25, 2011 at 1:12 PM, Dan Yost <yo...@gmail.com> wrote:
> Greetings,
>
> I'm at my wit's end trying to debug a troubling, highly irregular
> issue connecting to a SVN server via HTTPS. I am very confident this
> is a client issue, not a server issue. The server is CollabNet's
> Subversion Edge server, which has 1.6.15 built in. The client is
> 1.6.15 running under Cygwin. I have many, many clients, most of which
> are running Windows Server 2003 (if that matters)--all Cygwin, all
> this same package. Lots of traffic every day. Almost all of it works
> just fine.
>
> Now, a very important fact: the svn operations WORK 95% of the time.
> They just work. Everything is absolutely fine. This is an unattended,
> fully-automated environment, launched from cron (well, Task Scheduler
> when running on these Windows boxes). 95% of the time, the job
> launches, runs my "svn update" command from within one of its working
> copies:
>
> svn update --username=some --password=thing --non-interactive
>
>
> and it works beautifully.
>
> Every "once in awhile," for absolutely no reason I can find (have
> looked at logs, logs, logs, timing, the stars and planets,
> everything), with absolutely nothing that I can find having changed,
> it dies with this:
>
>
> svn: OPTIONS of 'https://myserver:9999/svn/some/path': Server
> certificate verification failed: certificate issued for a different
> hostname, issuer is not trusted (https://myserver:9999)
>
>
> (I have changed the paths/ports to protect the innocent).
>
>
> Now for most of my clients, the "issued for a different hostname" part
> is omitted--they still have the "issuer not trusted" part. That's
> because my one host that I'm really digging in to debug this has to
> use a different hostname for certain reasons. So, focus on the "issuer
> not trusted" part because that's what we see across all clients.
>
> But keep in mind: it works. It works 95% of the time. I change
> absolutely nothing. Having found no indication of why it will
> *randomly* fail with the above error, I decided to add the
> "--trust-cerver-cert" flag, thinking surely this will eliminate all
> woes. Just trust it and go (trust it like you already do 95% of the
> time anyway)!
>
>
> svn update --username=some --password=thing --non-interactive
> --trust-server-cert
>
>
> But, nope, I randomly get the very same error.
>
> These key facts:
>
> 1) It works 95% of the time (well, maybe 85%)
>
> 2) I can find NOTHING changing (on client or server) between a working
> op and a breaking op--it's all automated, and just randomly shanks
>
>
> are leading me to believe that this error message is actually a
> complete wild goose chase, and that the problem is not actually
> cert-based but is something else (with a wrong error message). In
> looking at the server HTTP logs, it does indeed appear that the client
> bails out before continuing the connection--there's no log, meaning it
> does appear that the client can't handshake (or refuses to), and then
> quits.
>
> I have thought about adding options to config (on the command line)
> such as ssl-trust-default-ca, but again, I already told svn to
> --trust-server-cert and it randomly still dies, so I don't think that
> will help.
>
> You'll be tempted to focus on SSL cert issues on the server or
> something, but remember that we don't change anything and it works
> fine almost all the time! (I'm still open to any thoughts of course).
>
> Can anybody point me to some other debug options, some other flag, or
> any other way to determine why, 15% of the time, it dies, yet 85% of
> the time it updates just fine?
>
> Thanks,
> Dan
>