You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by Zack Bartel <zb...@proofpoint.com> on 2021/01/05 22:49:20 UTC

Client / TS certificates 7.x vs 8.x

Hello,
We are using client -> TS SSL termination and using client certificates and a CA. While upgrading to 8.x this has stopped working and I'm having trouble figuring out why. I was wondering if there may have been some change in 8.x or error with my config  which would be obvious to someone.

The error is "SSL alert number 80" internal error.

SSL_connect:SSLv3/TLS read server certificate request
SSL_connect:SSLv3/TLS read server done
SSL_connect:SSLv3/TLS write client certificate
SSL_connect:SSLv3/TLS write client key exchange
SSL_connect:SSLv3/TLS write certificate verify
SSL_connect:SSLv3/TLS write change cipher spec


SSL_connect:SSLv3/TLS write finished
read from 0x564393d04170 [0x564393d0e6a3] (5 bytes => 5 (0x5))
0000 - 15 03 03 00 02                                    .....
read from 0x564393d04170 [0x564393d0e6a8] (2 bytes => 2 (0x2))
0000 - 02 50                                             .P

SSL3 alert read:fatal:internal error
SSL_connect:error in error
139713745085760:error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:../ssl/record/rec_layer_s3.c:1543:SSL alert number 80



I've inspected the traffic with Wireshark to confirm the exchange is identical between the versions up until the "Internal Error" response. Tried up to 7.1.12-rc0 which works and 8.0.0 -> 8.1.x which does not. I've also tried on Ubuntu and Centos8 with the same results.

Relevant config:


records.config:
CONFIG proxy.config.ssl.CA.cert.path STRING /ssl/
CONFIG proxy.config.ssl.CA.cert.filename STRING ca.pem

CONFIG proxy.config.ssl.client.verify.server INT 1
CONFIG proxy.config.ssl.client.certification_level INT 2
CONFIG proxy.config.ssl.client.CA.cert.path STRING /etc/pki/tls/certs/
CONFIG proxy.config.ssl.client.CA.cert.filename STRING ca-bundle.crt

CONFIG proxy.config.ssl.client.cipher_suite STRING TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA:AES256-SHA:DES-CBC3-SHA

CONFIG proxy.config.ssl.server.private_key.path STRING /ssl/
CONFIG proxy.config.ssl.server.cert.path STRING /ssl/

CONFIG proxy.config.ssl.server.cipher_suite STRING ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA

ssl_multicert.config:
ssl_cert_name=server.pem


Any help on this greatly appreciated, thank you,

Zack

Re: Client / TS certificates 7.x vs 8.x

Posted by Zack Bartel <zb...@proofpoint.com>.
Just to add more detail to this one. It turns out everything works great on the 9.0.0 build. Exploring upgrading directly to 9 and skipping 8 now.

Also, I tried recreating the CA, certs and keys but had the same result with 8.

On Jan 6, 2021, at 2:39 PM, Susan Hinrichs <sh...@verizonmedia.com>> wrote:

Looking at our production environment, I see that we use an absolute path in proxy.config.ssl.CA.cert.filename.  Could you try setting that to /ssl/ca.pem?  Perhaps a bug came in with the path manipulation.

On Wed, Jan 6, 2021 at 3:01 PM Zack Bartel <zb...@proofpoint.com>> wrote:
Thanks for the help!
Yes, the client certificate is directly signed by the certificate in ca.pem. There is no intermediary. I did not know about the "ssl" debug flag but this is very helpful. It seems your theory about failing the client cert verification was correct.

[Jan  6 08:21:22.635] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1213<https://urldefense.proofpoint.com/v2/url?u=http-3A__sslnetvconnection.cc-3A1213&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=NeldsD4eLgkhoyTV-xMLLQXlHwznW8HxcVoSqwV9Xj0&e=> (sslServerHandShakeEvent)> (ssl-diag) SSL handshake error: SSL_ERROR_WANT_READ (2), errno=0
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1139<https://urldefense.proofpoint.com/v2/url?u=http-3A__sslnetvconnection.cc-3A1139&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=2e8RrV4SJTDNVqVPg9Jqsd0kS3SFkE-W4WWwQbYSwdE&e=> (sslServerHandShakeEvent)> (ssl) Go on with the handshake state=5
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLUtils.cc:1510<https://urldefense.proofpoint.com/v2/url?u=http-3A__sslutils.cc-3A1510&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=Qw7t3wWHr5fpoD4lyUqEFXK7FOv3eHdmTYbq_Ppx84g&e=> (ssl_callback_info)> (ssl) ssl_callback_info ssl: 0x7ff228028ab0 where: 8193 ret: 1 State: SSLv3/TLS write server done
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLUtils.cc:401<https://urldefense.proofpoint.com/v2/url?u=http-3A__sslutils.cc-3A401&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=Vs4SLGP4rxroDg7Rbt7qkTKLaMA6Ya15ER4leCdjttY&e=> (ssl_verify_client_callback)> (ssl) Callback: verify client cert
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1603<https://urldefense.proofpoint.com/v2/url?u=http-3A__sslnetvconnection.cc-3A1603&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=KJEEYFXd7M33amuR5k3ehukmfIPWBLHNpjXtoK0VPf8&e=> (callHooks)> (ssl) callHooks sslHandshakeHookState=5
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1675<https://urldefense.proofpoint.com/v2/url?u=http-3A__sslnetvconnection.cc-3A1675&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=7E5pSrM1JfOjHcvAzZcn1oNDErDsn4l9_qZKkXOG3tE&e=> (callHooks)> (ssl) callHooks iterated to curHook=(nil)
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLUtils.cc:1510<https://urldefense.proofpoint.com/v2/url?u=http-3A__sslutils.cc-3A1510&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=Qw7t3wWHr5fpoD4lyUqEFXK7FOv3eHdmTYbq_Ppx84g&e=> (ssl_callback_info)> (ssl) ssl_callback_info ssl: 0x7ff228028ab0 where: 16392 ret: 592 State: error
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLUtils.cc:1510<https://urldefense.proofpoint.com/v2/url?u=http-3A__sslutils.cc-3A1510&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=Qw7t3wWHr5fpoD4lyUqEFXK7FOv3eHdmTYbq_Ppx84g&e=> (ssl_callback_info)> (ssl) ssl_callback_info ssl: 0x7ff228028ab0 where: 8194 ret: -1 State: error
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLUtils.cc:2344<https://urldefense.proofpoint.com/v2/url?u=http-3A__sslutils.cc-3A2344&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=bYSr4Zeblv-W2acIqCvJDwdjbv4Yd6hfhMfUKGV__Ls&e=> (SSLAccept)> (ssl.error.accept) SSL accept returned -1, ssl_error=1, ERR_get_error=337100934 (error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed)
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1213<https://urldefense.proofpoint.com/v2/url?u=http-3A__sslnetvconnection.cc-3A1213&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=NeldsD4eLgkhoyTV-xMLLQXlHwznW8HxcVoSqwV9Xj0&e=> (sslServerHandShakeEvent)> (ssl-diag) SSL::140678460933888:error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed:../ssl/statem/statem_srvr.c:3701: peer address is 192.168.1.4
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1213<https://urldefense.proofpoint.com/v2/url?u=http-3A__sslnetvconnection.cc-3A1213&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=NeldsD4eLgkhoyTV-xMLLQXlHwznW8HxcVoSqwV9Xj0&e=> (sslServerHandShakeEvent)> (ssl-diag) SSL handshake error: SSL_ERROR_SSL (1), errno=0
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1339<https://urldefense.proofpoint.com/v2/url?u=http-3A__sslnetvconnection.cc-3A1339&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=5R00uQMyJ9YmmMafXRE9xHQzKiK1yEVJf-X57vJ0jGg&e=> (sslServerHandShakeEvent)> (ssl-diag) SSLNetVConnection::sslServerHandShakeEvent, SSL_ERROR_SSL errno=0


The certificate is quite old but valid.
        Validity
            Not Before: Aug 18 14:29:55 2017 GMT
            Not After : Aug 16 14:29:55 2027 GMT

There is nothing noteworthy in diags.log nor any startup errors or warnings.

Thanks again,
Zack



On Jan 6, 2021, at 7:38 AM, Susan Hinrichs <sh...@verizonmedia.com>> wrote:

The config looks good to me.   There are autests that exercise ATS requiring certs from the client with a variety of CA verification configurations.  I haven't worked with the 8.x branch, but those tests are run against that branch I believe.

Is your client certificate directly signed by the certificate in ca.pem?  Is there an intermediate cert floating around someplace?  Assuming the certs are exchanged as in the working case, it seems likely that the failure is occurring in the client certificate verification.  I assume that ATS is sending the SSL alert.

Any interesting start up failure messages in diags.log?

The next thing to check is turning on debugging and setting the ssl debug tag.

On Tue, Jan 5, 2021 at 4:49 PM Zack Bartel <zb...@proofpoint.com>> wrote:
Hello,
We are using client -> TS SSL termination and using client certificates and a CA. While upgrading to 8.x this has stopped working and I'm having trouble figuring out why. I was wondering if there may have been some change in 8.x or error with my config  which would be obvious to someone.

The error is "SSL alert number 80" internal error.

SSL_connect:SSLv3/TLS read server certificate request
SSL_connect:SSLv3/TLS read server done
SSL_connect:SSLv3/TLS write client certificate
SSL_connect:SSLv3/TLS write client key exchange
SSL_connect:SSLv3/TLS write certificate verify
SSL_connect:SSLv3/TLS write change cipher spec


SSL_connect:SSLv3/TLS write finished
read from 0x564393d04170 [0x564393d0e6a3] (5 bytes => 5 (0x5))
0000 - 15 03 03 00 02                                    .....
read from 0x564393d04170 [0x564393d0e6a8] (2 bytes => 2 (0x2))
0000 - 02 50                                             .P

SSL3 alert read:fatal:internal error
SSL_connect:error in error
139713745085760:error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:../ssl/record/rec_layer_s3.c:1543:SSL alert number 80



I've inspected the traffic with Wireshark to confirm the exchange is identical between the versions up until the "Internal Error" response. Tried up to 7.1.12-rc0 which works and 8.0.0 -> 8.1.x which does not. I've also tried on Ubuntu and Centos8 with the same results.

Relevant config:


records.config:
CONFIG proxy.config.ssl.CA.cert.path STRING /ssl/
CONFIG proxy.config.ssl.CA.cert.filename STRING ca.pem

CONFIG proxy.config.ssl.client.verify.server INT 1
CONFIG proxy.config.ssl.client.certification_level INT 2
CONFIG proxy.config.ssl.client.CA.cert.path STRING /etc/pki/tls/certs/
CONFIG proxy.config.ssl.client.CA.cert.filename STRING ca-bundle.crt

CONFIG proxy.config.ssl.client.cipher_suite STRING TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA:AES256-SHA:DES-CBC3-SHA

CONFIG proxy.config.ssl.server.private_key.path STRING /ssl/
CONFIG proxy.config.ssl.server.cert.path STRING /ssl/

CONFIG proxy.config.ssl.server.cipher_suite STRING ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA

ssl_multicert.config:
ssl_cert_name=server.pem


Any help on this greatly appreciated, thank you,

Zack



Re: Client / TS certificates 7.x vs 8.x

Posted by Susan Hinrichs <sh...@verizonmedia.com>.
Looking at our production environment, I see that we use an absolute path
in proxy.config.ssl.CA.cert.filename.  Could you try setting that to
/ssl/ca.pem?  Perhaps a bug came in with the path manipulation.

On Wed, Jan 6, 2021 at 3:01 PM Zack Bartel <zb...@proofpoint.com> wrote:

> Thanks for the help!
> Yes, the client certificate is directly signed by the certificate in
> ca.pem. There is no intermediary. I did not know about the "ssl" debug flag
> but this is very helpful. It seems your theory about failing the client
> cert verification was correct.
>
> [Jan  6 08:21:22.635] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1213
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__sslnetvconnection.cc-3A1213&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=NeldsD4eLgkhoyTV-xMLLQXlHwznW8HxcVoSqwV9Xj0&e=>
> (sslServerHandShakeEvent)> (ssl-diag) SSL handshake error:
> SSL_ERROR_WANT_READ (2), errno=0
> [Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1139
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__sslnetvconnection.cc-3A1139&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=2e8RrV4SJTDNVqVPg9Jqsd0kS3SFkE-W4WWwQbYSwdE&e=>
> (sslServerHandShakeEvent)> (ssl) Go on with the handshake state=5
> [Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLUtils.cc:1510
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__sslutils.cc-3A1510&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=Qw7t3wWHr5fpoD4lyUqEFXK7FOv3eHdmTYbq_Ppx84g&e=>
> (ssl_callback_info)> (ssl) ssl_callback_info ssl: 0x7ff228028ab0 where:
> 8193 ret: 1 State: SSLv3/TLS write server done
> [Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLUtils.cc:401
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__sslutils.cc-3A401&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=Vs4SLGP4rxroDg7Rbt7qkTKLaMA6Ya15ER4leCdjttY&e=>
> (ssl_verify_client_callback)> (ssl) Callback: verify client cert
> [Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1603
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__sslnetvconnection.cc-3A1603&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=KJEEYFXd7M33amuR5k3ehukmfIPWBLHNpjXtoK0VPf8&e=>
> (callHooks)> (ssl) callHooks sslHandshakeHookState=5
> [Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1675
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__sslnetvconnection.cc-3A1675&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=7E5pSrM1JfOjHcvAzZcn1oNDErDsn4l9_qZKkXOG3tE&e=>
> (callHooks)> (ssl) callHooks iterated to curHook=(nil)
> [Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLUtils.cc:1510
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__sslutils.cc-3A1510&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=Qw7t3wWHr5fpoD4lyUqEFXK7FOv3eHdmTYbq_Ppx84g&e=>
> (ssl_callback_info)> (ssl) ssl_callback_info ssl: 0x7ff228028ab0 where:
> 16392 ret: 592 State: error
> [Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLUtils.cc:1510
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__sslutils.cc-3A1510&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=Qw7t3wWHr5fpoD4lyUqEFXK7FOv3eHdmTYbq_Ppx84g&e=>
> (ssl_callback_info)> (ssl) ssl_callback_info ssl: 0x7ff228028ab0 where:
> 8194 ret: -1 State: error
> [Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLUtils.cc:2344
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__sslutils.cc-3A2344&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=bYSr4Zeblv-W2acIqCvJDwdjbv4Yd6hfhMfUKGV__Ls&e=>
> (SSLAccept)> (ssl.error.accept) SSL accept returned -1, ssl_error=1,
> ERR_get_error=337100934 (error:1417C086:SSL
> routines:tls_process_client_certificate:certificate verify failed)
> [Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1213
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__sslnetvconnection.cc-3A1213&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=NeldsD4eLgkhoyTV-xMLLQXlHwznW8HxcVoSqwV9Xj0&e=>
> (sslServerHandShakeEvent)> (ssl-diag)
> SSL::140678460933888:error:1417C086:SSL
> routines:tls_process_client_certificate:certificate verify
> failed:../ssl/statem/statem_srvr.c:3701: peer address is 192.168.1.4
> [Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1213
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__sslnetvconnection.cc-3A1213&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=NeldsD4eLgkhoyTV-xMLLQXlHwznW8HxcVoSqwV9Xj0&e=>
> (sslServerHandShakeEvent)> (ssl-diag) SSL handshake error: SSL_ERROR_SSL
> (1), errno=0
> [Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1339
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__sslnetvconnection.cc-3A1339&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=5R00uQMyJ9YmmMafXRE9xHQzKiK1yEVJf-X57vJ0jGg&e=>
> (sslServerHandShakeEvent)> (ssl-diag)
> SSLNetVConnection::sslServerHandShakeEvent, SSL_ERROR_SSL errno=0
>
>
> The certificate is quite old but valid.
>         Validity
>             Not Before: Aug 18 14:29:55 2017 GMT
>             Not After : Aug 16 14:29:55 2027 GMT
>
> There is nothing noteworthy in diags.log nor any startup errors or
> warnings.
>
> Thanks again,
> Zack
>
>
>
> On Jan 6, 2021, at 7:38 AM, Susan Hinrichs <sh...@verizonmedia.com>
> wrote:
>
> The config looks good to me.   There are autests that exercise ATS
> requiring certs from the client with a variety of CA verification
> configurations.  I haven't worked with the 8.x branch, but those tests are
> run against that branch I believe.
>
> Is your client certificate directly signed by the certificate in ca.pem?
> Is there an intermediate cert floating around someplace?  Assuming the
> certs are exchanged as in the working case, it seems likely that the
> failure is occurring in the client certificate verification.  I assume that
> ATS is sending the SSL alert.
>
> Any interesting start up failure messages in diags.log?
>
> The next thing to check is turning on debugging and setting the ssl debug
> tag.
>
> On Tue, Jan 5, 2021 at 4:49 PM Zack Bartel <zb...@proofpoint.com> wrote:
>
>> Hello,
>> We are using client -> TS SSL termination and using client certificates
>> and a CA. While upgrading to 8.x this has stopped working and I'm having
>> trouble figuring out why. I was wondering if there may have been some
>> change in 8.x or error with my config  which would be obvious to someone.
>>
>> The error is "SSL alert number 80" internal error.
>>
>> SSL_connect:SSLv3/TLS read server certificate request
>> SSL_connect:SSLv3/TLS read server done
>> SSL_connect:SSLv3/TLS write client certificate
>> SSL_connect:SSLv3/TLS write client key exchange
>> SSL_connect:SSLv3/TLS write certificate verify
>> SSL_connect:SSLv3/TLS write change cipher spec
>>
>>
>> SSL_connect:SSLv3/TLS write finished
>> read from 0x564393d04170 [0x564393d0e6a3] (5 bytes => 5 (0x5))
>> 0000 - 15 03 03 00 02                                    .....
>> read from 0x564393d04170 [0x564393d0e6a8] (2 bytes => 2 (0x2))
>> 0000 - 02 50                                             .P
>>
>> *SSL3 alert read:fatal:internal error*
>> *SSL_connect:error in error*
>> *139713745085760:error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert
>> internal error:../ssl/record/rec_layer_s3.c:1543:SSL alert number 80*
>>
>>
>>
>> I've inspected the traffic with Wireshark to confirm the exchange is
>> identical between the versions up until the "Internal Error" response.
>> Tried up to 7.1.12-rc0 which works and 8.0.0 -> 8.1.x which does not. I've
>> also tried on Ubuntu and Centos8 with the same results.
>>
>> Relevant config:
>>
>>
>> records.config:
>> CONFIG proxy.config.ssl.CA.cert.path STRING /ssl/
>> CONFIG proxy.config.ssl.CA.cert.filename STRING ca.pem
>>
>> CONFIG proxy.config.ssl.client.verify.server INT 1
>> CONFIG proxy.config.ssl.client.certification_level INT 2
>> CONFIG proxy.config.ssl.client.CA.cert.path STRING /etc/pki/tls/certs/
>> CONFIG proxy.config.ssl.client.CA.cert.filename STRING ca-bundle.crt
>>
>> CONFIG proxy.config.ssl.client.cipher_suite STRING
>> TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA:AES256-SHA:DES-CBC3-SHA
>>
>> CONFIG proxy.config.ssl.server.private_key.path STRING /ssl/
>> CONFIG proxy.config.ssl.server.cert.path STRING /ssl/
>>
>> CONFIG proxy.config.ssl.server.cipher_suite STRING
>> ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
>>
>> ssl_multicert.config:
>> ssl_cert_name=server.pem
>>
>>
>> Any help on this greatly appreciated, thank you,
>>
>> Zack
>>
>
>

Re: Client / TS certificates 7.x vs 8.x

Posted by Zack Bartel <zb...@proofpoint.com>.
Thanks for the help!
Yes, the client certificate is directly signed by the certificate in ca.pem. There is no intermediary. I did not know about the "ssl" debug flag but this is very helpful. It seems your theory about failing the client cert verification was correct.

[Jan  6 08:21:22.635] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1213<http://sslnetvconnection.cc:1213> (sslServerHandShakeEvent)> (ssl-diag) SSL handshake error: SSL_ERROR_WANT_READ (2), errno=0
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1139<http://sslnetvconnection.cc:1139> (sslServerHandShakeEvent)> (ssl) Go on with the handshake state=5
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLUtils.cc:1510<http://sslutils.cc:1510> (ssl_callback_info)> (ssl) ssl_callback_info ssl: 0x7ff228028ab0 where: 8193 ret: 1 State: SSLv3/TLS write server done
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLUtils.cc:401<http://sslutils.cc:401> (ssl_verify_client_callback)> (ssl) Callback: verify client cert
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1603<http://sslnetvconnection.cc:1603> (callHooks)> (ssl) callHooks sslHandshakeHookState=5
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1675<http://sslnetvconnection.cc:1675> (callHooks)> (ssl) callHooks iterated to curHook=(nil)
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLUtils.cc:1510<http://sslutils.cc:1510> (ssl_callback_info)> (ssl) ssl_callback_info ssl: 0x7ff228028ab0 where: 16392 ret: 592 State: error
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLUtils.cc:1510<http://sslutils.cc:1510> (ssl_callback_info)> (ssl) ssl_callback_info ssl: 0x7ff228028ab0 where: 8194 ret: -1 State: error
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLUtils.cc:2344<http://sslutils.cc:2344> (SSLAccept)> (ssl.error.accept) SSL accept returned -1, ssl_error=1, ERR_get_error=337100934 (error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed)
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1213<http://sslnetvconnection.cc:1213> (sslServerHandShakeEvent)> (ssl-diag) SSL::140678460933888:error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed:../ssl/statem/statem_srvr.c:3701: peer address is 192.168.1.4
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1213<http://sslnetvconnection.cc:1213> (sslServerHandShakeEvent)> (ssl-diag) SSL handshake error: SSL_ERROR_SSL (1), errno=0
[Jan  6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1339<http://sslnetvconnection.cc:1339> (sslServerHandShakeEvent)> (ssl-diag) SSLNetVConnection::sslServerHandShakeEvent, SSL_ERROR_SSL errno=0


The certificate is quite old but valid.
        Validity
            Not Before: Aug 18 14:29:55 2017 GMT
            Not After : Aug 16 14:29:55 2027 GMT

There is nothing noteworthy in diags.log nor any startup errors or warnings.

Thanks again,
Zack



On Jan 6, 2021, at 7:38 AM, Susan Hinrichs <sh...@verizonmedia.com>> wrote:

The config looks good to me.   There are autests that exercise ATS requiring certs from the client with a variety of CA verification configurations.  I haven't worked with the 8.x branch, but those tests are run against that branch I believe.

Is your client certificate directly signed by the certificate in ca.pem?  Is there an intermediate cert floating around someplace?  Assuming the certs are exchanged as in the working case, it seems likely that the failure is occurring in the client certificate verification.  I assume that ATS is sending the SSL alert.

Any interesting start up failure messages in diags.log?

The next thing to check is turning on debugging and setting the ssl debug tag.

On Tue, Jan 5, 2021 at 4:49 PM Zack Bartel <zb...@proofpoint.com>> wrote:
Hello,
We are using client -> TS SSL termination and using client certificates and a CA. While upgrading to 8.x this has stopped working and I'm having trouble figuring out why. I was wondering if there may have been some change in 8.x or error with my config  which would be obvious to someone.

The error is "SSL alert number 80" internal error.

SSL_connect:SSLv3/TLS read server certificate request
SSL_connect:SSLv3/TLS read server done
SSL_connect:SSLv3/TLS write client certificate
SSL_connect:SSLv3/TLS write client key exchange
SSL_connect:SSLv3/TLS write certificate verify
SSL_connect:SSLv3/TLS write change cipher spec


SSL_connect:SSLv3/TLS write finished
read from 0x564393d04170 [0x564393d0e6a3] (5 bytes => 5 (0x5))
0000 - 15 03 03 00 02                                    .....
read from 0x564393d04170 [0x564393d0e6a8] (2 bytes => 2 (0x2))
0000 - 02 50                                             .P

SSL3 alert read:fatal:internal error
SSL_connect:error in error
139713745085760:error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:../ssl/record/rec_layer_s3.c:1543:SSL alert number 80



I've inspected the traffic with Wireshark to confirm the exchange is identical between the versions up until the "Internal Error" response. Tried up to 7.1.12-rc0 which works and 8.0.0 -> 8.1.x which does not. I've also tried on Ubuntu and Centos8 with the same results.

Relevant config:


records.config:
CONFIG proxy.config.ssl.CA.cert.path STRING /ssl/
CONFIG proxy.config.ssl.CA.cert.filename STRING ca.pem

CONFIG proxy.config.ssl.client.verify.server INT 1
CONFIG proxy.config.ssl.client.certification_level INT 2
CONFIG proxy.config.ssl.client.CA.cert.path STRING /etc/pki/tls/certs/
CONFIG proxy.config.ssl.client.CA.cert.filename STRING ca-bundle.crt

CONFIG proxy.config.ssl.client.cipher_suite STRING TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA:AES256-SHA:DES-CBC3-SHA

CONFIG proxy.config.ssl.server.private_key.path STRING /ssl/
CONFIG proxy.config.ssl.server.cert.path STRING /ssl/

CONFIG proxy.config.ssl.server.cipher_suite STRING ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA

ssl_multicert.config:
ssl_cert_name=server.pem


Any help on this greatly appreciated, thank you,

Zack


Re: Client / TS certificates 7.x vs 8.x

Posted by Susan Hinrichs <sh...@verizonmedia.com>.
The config looks good to me.   There are autests that exercise ATS
requiring certs from the client with a variety of CA verification
configurations.  I haven't worked with the 8.x branch, but those tests are
run against that branch I believe.

Is your client certificate directly signed by the certificate in ca.pem?
Is there an intermediate cert floating around someplace?  Assuming the
certs are exchanged as in the working case, it seems likely that the
failure is occurring in the client certificate verification.  I assume that
ATS is sending the SSL alert.

Any interesting start up failure messages in diags.log?

The next thing to check is turning on debugging and setting the ssl debug
tag.

On Tue, Jan 5, 2021 at 4:49 PM Zack Bartel <zb...@proofpoint.com> wrote:

> Hello,
> We are using client -> TS SSL termination and using client certificates
> and a CA. While upgrading to 8.x this has stopped working and I'm having
> trouble figuring out why. I was wondering if there may have been some
> change in 8.x or error with my config  which would be obvious to someone.
>
> The error is "SSL alert number 80" internal error.
>
> SSL_connect:SSLv3/TLS read server certificate request
> SSL_connect:SSLv3/TLS read server done
> SSL_connect:SSLv3/TLS write client certificate
> SSL_connect:SSLv3/TLS write client key exchange
> SSL_connect:SSLv3/TLS write certificate verify
> SSL_connect:SSLv3/TLS write change cipher spec
>
>
> SSL_connect:SSLv3/TLS write finished
> read from 0x564393d04170 [0x564393d0e6a3] (5 bytes => 5 (0x5))
> 0000 - 15 03 03 00 02                                    .....
> read from 0x564393d04170 [0x564393d0e6a8] (2 bytes => 2 (0x2))
> 0000 - 02 50                                             .P
>
> *SSL3 alert read:fatal:internal error*
> *SSL_connect:error in error*
> *139713745085760:error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert
> internal error:../ssl/record/rec_layer_s3.c:1543:SSL alert number 80*
>
>
>
> I've inspected the traffic with Wireshark to confirm the exchange is
> identical between the versions up until the "Internal Error" response.
> Tried up to 7.1.12-rc0 which works and 8.0.0 -> 8.1.x which does not. I've
> also tried on Ubuntu and Centos8 with the same results.
>
> Relevant config:
>
>
> records.config:
> CONFIG proxy.config.ssl.CA.cert.path STRING /ssl/
> CONFIG proxy.config.ssl.CA.cert.filename STRING ca.pem
>
> CONFIG proxy.config.ssl.client.verify.server INT 1
> CONFIG proxy.config.ssl.client.certification_level INT 2
> CONFIG proxy.config.ssl.client.CA.cert.path STRING /etc/pki/tls/certs/
> CONFIG proxy.config.ssl.client.CA.cert.filename STRING ca-bundle.crt
>
> CONFIG proxy.config.ssl.client.cipher_suite STRING
> TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA:AES256-SHA:DES-CBC3-SHA
>
> CONFIG proxy.config.ssl.server.private_key.path STRING /ssl/
> CONFIG proxy.config.ssl.server.cert.path STRING /ssl/
>
> CONFIG proxy.config.ssl.server.cipher_suite STRING
> ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
>
> ssl_multicert.config:
> ssl_cert_name=server.pem
>
>
> Any help on this greatly appreciated, thank you,
>
> Zack
>