You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Sandy <sa...@gmail.com> on 2017/11/13 14:54:32 UTC
Facing issues in enabling two way ssl in ambari for server-agent communication
We are trying to setup security in server agent communication using
instructions provided in this
https://community.hortonworks.com/articles/107092/configure-2-way-ssl-between-ambari-server-and-amba.html
Here are the things I've done:
1. Obtained Certificates from CA for both server and agent machines.
2. Placed them in corresponding directories in both server and agent.
3. When we try to manually verify authentication using following command,
two way ssl seems to be working (*openssl s_client -cert
agent-hostname.crt -key **agent-hostname**.key
-CAfile /var/lib/ambari-agent/keys/ca.crt -connect server-hostname:8441
-msg) Detailed logs below*
4. But ambari agent throws following error
INFO 2017-11-13 14:36:43,411 NetUtil.py:70 - Connecting to
https://server-hostname:8440/connection_info
INFO 2017-11-13 14:36:43,426 security.py:55 - Server require two-way SSL
authentication. Use it instead of one-way...
INFO 2017-11-13 14:36:43,426 security.py:183 - Server certicate exists, ok
INFO 2017-11-13 14:36:43,426 security.py:191 - Agent key exists, ok
INFO 2017-11-13 14:36:43,427 security.py:199 - Agent certificate exists, ok
INFO 2017-11-13 14:36:43,427 security.py:94 - SSL Connect being called..
connecting to the server
ERROR 2017-11-13 14:36:43,432 security.py:81 - Two-way SSL authentication
failed. Ensure that server and agent certificates were signed by the same
CA and restart the agent.
In order to receive a new agent certificate, remove existing certificate
file from keys directory. As a workaround you can turn off two-way SSL
authentication in server configuration(ambari.properties)
Exiting..
ERROR 2017-11-13 14:36:43,432 Controller.py:226 - Unable to connect to:
https://server-hostname:8441/agent/v1/register/agent-hostname
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line
175, in registerWithServer
ret = self.sendRequest(self.registerUrl, data)
File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line
549, in sendRequest
raise IOError('Request to {0} failed due to {1}'.format(url, str
(exception)))
IOError: Request to
https://server-hostname:8441/agent/v1/register/agent-hostname failed due to
[Errno 8] _ssl.c:492: EOF occurred in violation of protocol
ERROR 2017-11-13 14:36:43,433 Controller.py:227 - Error:Request to
https://server-hostname:8441/agent/v1/register/agent-hostname failed due to
[Errno 8] _ssl.c:492: EOF occurred in violation of protocol
WARNING 2017-11-13 14:36:43,433 Controller.py:228 - Sleeping for 11
seconds and then trying again
*Can someone help ?*
-bash-4.1$ *openssl s_client -cert agent-hostname.crt -key
**agent-hostname**.key
-CAfile /var/lib/ambari-agent/keys/ca.crt -connect server-hostname:8441
-msg *
CONNECTED(00000003)
>>> TLS 1.2 Handshake [length 00f2], ClientHello
...
<<< TLS 1.2 Handshake [length 0051], ServerHello
...
<<< TLS 1.2 Handshake [length 0524], Certificate
...
verify return:1
<<< TLS 1.2 Handshake [length 0191], ServerKeyExchange
...
<<< TLS 1.2 Handshake [length 00d2], CertificateRequest
...
<<< TLS 1.2 Handshake [length 0004], ServerHelloDone
0e 00 00 00
>>> TLS 1.2 Handshake [length 0fe2], Certificate
...
>>> TLS 1.2 Handshake [length 008a], ClientKeyExchange
...
>>> TLS 1.2 Handshake [length 0108], CertificateVerify
...
>>> TLS 1.2 ChangeCipherSpec [length 0001]
01
>>> TLS 1.2 Handshake [length 0010], Finished
---
Certificate chain
---
Server certificate
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
...
---
Acceptable client certificate CA names
---
SSL handshake has read 2017 bytes and written 4534 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID:
5A09AF66C19A54A200221F9EFACC20642DBDCCE50099EE6836FDA0B4ECE33EF6
Session-ID-ctx:
Master-Key:
F4BD9CEA03E292AC4DC696B46E3CD1348BD954C300FAE6A07697507937B422187B51FB0814B20CFBCAFD21A65B30BEBC
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1510584166
Timeout : 300 (sec)
* Verify return code: 0 (ok) *
---
-bash-4.1$
--
*Sandeep Kumar,*
Mobile +91-9866507368
*“Happiness is not a destination, It is the journey”*
Re: Facing issues in enabling two way ssl in ambari for server-agent communication
Posted by Sandy <sa...@gmail.com>.
Could dig bit deeper into the issue and found one resource online where a
fix is suggested to add following into ambari-agent.ini
force_https_protocol=PROTOCOL_TLSv1_2 (source:
http://knowledge.teradata.com/KCS/id/KCS008843)
On doing so, I am getting following error:
INFO 2017-11-20 10:18:39,540 NetUtil.py:70 - Connecting to
https://ambari-server-host:8440/ca
WARNING 2017-11-20 10:18:39,541 NetUtil.py:101 - Failed to connect to
https://ambari-server-host:8440/ca due to 'module' object has no attribute
'PROTOCOL_TLSv1_2'
WARNING 2017-11-20 10:18:39,541 NetUtil.py:124 - Server at
https://ambari-server-host:8440 is not reachable, sleeping for 10 seconds...
INFO 2017-11-20 10:18:49,541 NetUtil.py:70 - Connecting to
https://ambari-server-host:8440/ca
*I am using python 2.6.6 and it seems ssl.PROTOCOL_TLSv1_2 was added in
python 2.7.9*
Using openssl s_client, I am successfully able to connect to the server and
get certificate.
*openssl s_client -connect ambari-server-host:8441 -tls1_2*
7iAh9/YddjuVxLzd1wLhpoEDcGPNj6HFkSu/zGucYV2F3eshEdIIqDFiE177styv
3og0SjEvNuIwa1ECYr+9Qm5yYk82NAowf5cQcx+ykboEsFMfBkvmmw0JhZMp1OB5
qnYL1cGI4Sp55w==
-----END CERTIFICATE-----
...
---
...
---
SSL handshake has read 3184 bytes and written 338 bytes
---
New, TLSv1/SSLv3, Cipher is DES-CBC3-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.1
Cipher : DES-CBC3-SHA
Session-ID:
5A12BA5D9A2296C2B43BD1869D2DEFD97B334D0A07C843613206D66AACA7E99A
Session-ID-ctx:
Master-Key:
AA96C013DE644728F03D994B318A5785760DFADAB622323A21E8D593F41485943495B810C78CA246A05A892012BEA11D
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1511176797
Timeout : 7200 (sec)
Verify return code: 0 (ok)
But agent (python code) fails to do so [logs snippet below]
INFO 2017-11-20 10:39:29,799 security.py:93 - SSL Connect being called..
connecting to the server
ERROR 2017-11-20 10:39:29,815 security.py:80 - Two-way SSL authentication
failed. Ensure that server and agent certificates were signed by the same
CA and restart the agent.
In order to receive a new agent certificate, remove existing certificate
file from keys directory. As a workaround you can turn off two-way SSL
authentication in server configuration(ambari.properties)
Exiting..
ERROR 2017-11-20 10:39:29,815 Controller.py:226 - Unable to connect to:
https://ambari-server-host:8441/agent/v1/register/ambari-agent-host
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line
175, in registerWithServer
ret = self.sendRequest(self.registerUrl, data)
File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line
549, in sendRequest
raise IOError('Request to {0} failed due to {1}'.format(url,
str(exception)))
*IOError: Request to
https://ambari-server-host:8441/agent/v1/register/ambari-agent-host
<https://ambari-server-host:8441/agent/v1/register/ambari-agent-host>
failed due to [Errno 1] _ssl.c:492: error:14090086:SSL
routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed*
*ERROR 2017-11-20 10:39:29,816 Controller.py:227 - Error:Request to
https://ambari-server-host:8441/agent/v1/register/ambari-agent-host
<https://ambari-server-host:8441/agent/v1/register/ambari-agent-host>
failed due to [Errno 1] _ssl.c:492: error:14090086:SSL
routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed*
WARNING 2017-11-20 10:39:29,816 Controller.py:228 - Sleeping for 7 seconds
and then trying again
Currently, *It's not possible for us to have python 2.7.9 installed on our
production machines*. Is there any other way/alternative to move past this
problem? and Use Two-Way SSL in production.
Regards,
Sandeep.
On Tue, Nov 14, 2017 at 8:12 AM, Sandy <sa...@gmail.com> wrote:
> Yeah. I had put Root CA cert CA.crt in server keys directory and added
> agent cert to to keystore too.
>
> Followed the exact instructions given in link above.
>
> I'm using just two machine setup. One server and one agent machine.
>
> Sandeep.
>
>
> On Mon, 13 Nov 2017 at 9:08 PM, Gonzalo Herreros <gh...@gmail.com>
> wrote:
>
>> Did you also put the ca.crt file on the server
>> /var/lib/ambari-server/keys?
>> Also in the steps you don't mention adding all the certs to the server
>> keystore which is in the instructions
>>
>> I think it's much less error prone to let ambari create and assign the
>> certs it uses to communicate with agents (being careful to configure
>> longer
>> life)
>>
>>
>> On 13 November 2017 at 14:54, Sandy <sa...@gmail.com> wrote:
>>
>> > We are trying to setup security in server agent communication using
>> > instructions provided in this
>> >
>> > https://community.hortonworks.com/articles/107092/configure-
>> > 2-way-ssl-between-ambari-server-and-amba.html
>> >
>> > Here are the things I've done:
>> > 1. Obtained Certificates from CA for both server and agent machines.
>> > 2. Placed them in corresponding directories in both server and agent.
>> > 3. When we try to manually verify authentication using following
>> command,
>> > two way ssl seems to be working (*openssl s_client -cert
>> > agent-hostname.crt -key **agent-hostname**.key
>> > -CAfile /var/lib/ambari-agent/keys/ca.crt -connect server-hostname:8441
>> > -msg) Detailed logs below*
>> > 4. But ambari agent throws following error
>> >
>> > INFO 2017-11-13 14:36:43,411 NetUtil.py:70 - Connecting to
>> > https://server-hostname:8440/connection_info
>> >
>> > INFO 2017-11-13 14:36:43,426 security.py:55 - Server require two-way SSL
>> > authentication. Use it instead of one-way...
>> >
>> > INFO 2017-11-13 14:36:43,426 security.py:183 - Server certicate exists,
>> ok
>> >
>> > INFO 2017-11-13 14:36:43,426 security.py:191 - Agent key exists, ok
>> >
>> > INFO 2017-11-13 14:36:43,427 security.py:199 - Agent certificate
>> exists, ok
>> >
>> > INFO 2017-11-13 14:36:43,427 security.py:94 - SSL Connect being called..
>> > connecting to the server
>> >
>> > ERROR 2017-11-13 14:36:43,432 security.py:81 - Two-way SSL
>> authentication
>> > failed. Ensure that server and agent certificates were signed by the
>> same
>> > CA and restart the agent.
>> >
>> > In order to receive a new agent certificate, remove existing certificate
>> > file from keys directory. As a workaround you can turn off two-way SSL
>> > authentication in server configuration(ambari.properties)
>> >
>> > Exiting..
>> >
>> > ERROR 2017-11-13 14:36:43,432 Controller.py:226 - Unable to connect to:
>> > https://server-hostname:8441/agent/v1/register/agent-hostname
>> >
>> > Traceback (most recent call last):
>> >
>> > File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py",
>> line
>> > 175, in registerWithServer
>> >
>> > ret = self.sendRequest(self.registerUrl, data)
>> >
>> > File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py",
>> line
>> > 549, in sendRequest
>> >
>> > raise IOError('Request to {0} failed due to {1}'.format(url, str
>> > (exception)))
>> >
>> > IOError: Request to
>> > https://server-hostname:8441/agent/v1/register/agent-hostname failed
>> due
>> > to
>> > [Errno 8] _ssl.c:492: EOF occurred in violation of protocol
>> >
>> > ERROR 2017-11-13 14:36:43,433 Controller.py:227 - Error:Request to
>> > https://server-hostname:8441/agent/v1/register/agent-hostname failed
>> due
>> > to
>> > [Errno 8] _ssl.c:492: EOF occurred in violation of protocol
>> >
>> > WARNING 2017-11-13 14:36:43,433 Controller.py:228 - Sleeping for 11
>> > seconds and then trying again
>> >
>> >
>> > *Can someone help ?*
>> >
>> > -bash-4.1$ *openssl s_client -cert agent-hostname.crt -key
>> > **agent-hostname**.key
>> > -CAfile /var/lib/ambari-agent/keys/ca.crt -connect server-hostname:8441
>> > -msg *
>> >
>> > CONNECTED(00000003)
>> >
>> > >>> TLS 1.2 Handshake [length 00f2], ClientHello
>> >
>> > ...
>> >
>> > <<< TLS 1.2 Handshake [length 0051], ServerHello
>> >
>> > ...
>> >
>> > <<< TLS 1.2 Handshake [length 0524], Certificate
>> >
>> > ...
>> >
>> > verify return:1
>> >
>> > <<< TLS 1.2 Handshake [length 0191], ServerKeyExchange
>> >
>> > ...
>> >
>> > <<< TLS 1.2 Handshake [length 00d2], CertificateRequest
>> >
>> > ...
>> >
>> > <<< TLS 1.2 Handshake [length 0004], ServerHelloDone
>> >
>> > 0e 00 00 00
>> >
>> > >>> TLS 1.2 Handshake [length 0fe2], Certificate
>> >
>> > ...
>> >
>> > >>> TLS 1.2 Handshake [length 008a], ClientKeyExchange
>> >
>> > ...
>> >
>> > >>> TLS 1.2 Handshake [length 0108], CertificateVerify
>> >
>> > ...
>> >
>> > >>> TLS 1.2 ChangeCipherSpec [length 0001]
>> >
>> > 01
>> >
>> > >>> TLS 1.2 Handshake [length 0010], Finished
>> >
>> > ---
>> >
>> > Certificate chain
>> >
>> > ---
>> >
>> > Server certificate
>> >
>> > -----BEGIN CERTIFICATE-----
>> >
>> > ...
>> >
>> > -----END CERTIFICATE-----
>> >
>> > ...
>> >
>> > ---
>> >
>> > Acceptable client certificate CA names
>> >
>> > ---
>> >
>> > SSL handshake has read 2017 bytes and written 4534 bytes
>> >
>> > ---
>> >
>> > New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
>> >
>> > Server public key is 2048 bit
>> >
>> > Secure Renegotiation IS supported
>> >
>> > Compression: NONE
>> >
>> > Expansion: NONE
>> >
>> > SSL-Session:
>> >
>> > Protocol : TLSv1.2
>> >
>> > Cipher : ECDHE-RSA-AES256-GCM-SHA384
>> >
>> > Session-ID:
>> > 5A09AF66C19A54A200221F9EFACC20642DBDCCE50099EE6836FDA0B4ECE33EF6
>> >
>> > Session-ID-ctx:
>> >
>> > Master-Key:
>> > F4BD9CEA03E292AC4DC696B46E3CD1348BD954C300FAE6A07697507937B4
>> > 22187B51FB0814B20CFBCAFD21A65B30BEBC
>> >
>> >
>> > Key-Arg : None
>> >
>> > Krb5 Principal: None
>> >
>> > PSK identity: None
>> >
>> > PSK identity hint: None
>> >
>> > Start Time: 1510584166
>> >
>> > Timeout : 300 (sec)
>> >
>> > * Verify return code: 0 (ok) *
>> >
>> > ---
>> >
>> > -bash-4.1$
>> >
>> > --
>> >
>> > *Sandeep Kumar,*
>> > Mobile +91-9866507368 <+91%2098665%2007368>
>> >
>> > *“Happiness is not a destination, It is the journey”*
>> >
>>
> --
>
> *Sandeep Kumar,*
> Mobile +91-9866507368 <+91%2098665%2007368>
>
> *“Happiness is not a destination, It is the journey”*
>
>
--
*Sandeep Kumar,*
Mobile +91-9866507368
*“Happiness is not a destination, It is the journey”*
Re: Facing issues in enabling two way ssl in ambari for server-agent communication
Posted by Sandy <sa...@gmail.com>.
Yeah. I had put Root CA cert CA.crt in server keys directory and added
agent cert to to keystore too.
Followed the exact instructions given in link above.
I'm using just two machine setup. One server and one agent machine.
Sandeep.
On Mon, 13 Nov 2017 at 9:08 PM, Gonzalo Herreros <gh...@gmail.com>
wrote:
> Did you also put the ca.crt file on the server /var/lib/ambari-server/keys?
> Also in the steps you don't mention adding all the certs to the server
> keystore which is in the instructions
>
> I think it's much less error prone to let ambari create and assign the
> certs it uses to communicate with agents (being careful to configure longer
> life)
>
>
> On 13 November 2017 at 14:54, Sandy <sa...@gmail.com> wrote:
>
> > We are trying to setup security in server agent communication using
> > instructions provided in this
> >
> > https://community.hortonworks.com/articles/107092/configure-
> > 2-way-ssl-between-ambari-server-and-amba.html
> >
> > Here are the things I've done:
> > 1. Obtained Certificates from CA for both server and agent machines.
> > 2. Placed them in corresponding directories in both server and agent.
> > 3. When we try to manually verify authentication using following
> command,
> > two way ssl seems to be working (*openssl s_client -cert
> > agent-hostname.crt -key **agent-hostname**.key
> > -CAfile /var/lib/ambari-agent/keys/ca.crt -connect server-hostname:8441
> > -msg) Detailed logs below*
> > 4. But ambari agent throws following error
> >
> > INFO 2017-11-13 14:36:43,411 NetUtil.py:70 - Connecting to
> > https://server-hostname:8440/connection_info
> >
> > INFO 2017-11-13 14:36:43,426 security.py:55 - Server require two-way SSL
> > authentication. Use it instead of one-way...
> >
> > INFO 2017-11-13 14:36:43,426 security.py:183 - Server certicate exists,
> ok
> >
> > INFO 2017-11-13 14:36:43,426 security.py:191 - Agent key exists, ok
> >
> > INFO 2017-11-13 14:36:43,427 security.py:199 - Agent certificate exists,
> ok
> >
> > INFO 2017-11-13 14:36:43,427 security.py:94 - SSL Connect being called..
> > connecting to the server
> >
> > ERROR 2017-11-13 14:36:43,432 security.py:81 - Two-way SSL authentication
> > failed. Ensure that server and agent certificates were signed by the same
> > CA and restart the agent.
> >
> > In order to receive a new agent certificate, remove existing certificate
> > file from keys directory. As a workaround you can turn off two-way SSL
> > authentication in server configuration(ambari.properties)
> >
> > Exiting..
> >
> > ERROR 2017-11-13 14:36:43,432 Controller.py:226 - Unable to connect to:
> > https://server-hostname:8441/agent/v1/register/agent-hostname
> >
> > Traceback (most recent call last):
> >
> > File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py",
> line
> > 175, in registerWithServer
> >
> > ret = self.sendRequest(self.registerUrl, data)
> >
> > File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py",
> line
> > 549, in sendRequest
> >
> > raise IOError('Request to {0} failed due to {1}'.format(url, str
> > (exception)))
> >
> > IOError: Request to
> > https://server-hostname:8441/agent/v1/register/agent-hostname failed due
> > to
> > [Errno 8] _ssl.c:492: EOF occurred in violation of protocol
> >
> > ERROR 2017-11-13 14:36:43,433 Controller.py:227 - Error:Request to
> > https://server-hostname:8441/agent/v1/register/agent-hostname failed due
> > to
> > [Errno 8] _ssl.c:492: EOF occurred in violation of protocol
> >
> > WARNING 2017-11-13 14:36:43,433 Controller.py:228 - Sleeping for 11
> > seconds and then trying again
> >
> >
> > *Can someone help ?*
> >
> > -bash-4.1$ *openssl s_client -cert agent-hostname.crt -key
> > **agent-hostname**.key
> > -CAfile /var/lib/ambari-agent/keys/ca.crt -connect server-hostname:8441
> > -msg *
> >
> > CONNECTED(00000003)
> >
> > >>> TLS 1.2 Handshake [length 00f2], ClientHello
> >
> > ...
> >
> > <<< TLS 1.2 Handshake [length 0051], ServerHello
> >
> > ...
> >
> > <<< TLS 1.2 Handshake [length 0524], Certificate
> >
> > ...
> >
> > verify return:1
> >
> > <<< TLS 1.2 Handshake [length 0191], ServerKeyExchange
> >
> > ...
> >
> > <<< TLS 1.2 Handshake [length 00d2], CertificateRequest
> >
> > ...
> >
> > <<< TLS 1.2 Handshake [length 0004], ServerHelloDone
> >
> > 0e 00 00 00
> >
> > >>> TLS 1.2 Handshake [length 0fe2], Certificate
> >
> > ...
> >
> > >>> TLS 1.2 Handshake [length 008a], ClientKeyExchange
> >
> > ...
> >
> > >>> TLS 1.2 Handshake [length 0108], CertificateVerify
> >
> > ...
> >
> > >>> TLS 1.2 ChangeCipherSpec [length 0001]
> >
> > 01
> >
> > >>> TLS 1.2 Handshake [length 0010], Finished
> >
> > ---
> >
> > Certificate chain
> >
> > ---
> >
> > Server certificate
> >
> > -----BEGIN CERTIFICATE-----
> >
> > ...
> >
> > -----END CERTIFICATE-----
> >
> > ...
> >
> > ---
> >
> > Acceptable client certificate CA names
> >
> > ---
> >
> > SSL handshake has read 2017 bytes and written 4534 bytes
> >
> > ---
> >
> > New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
> >
> > Server public key is 2048 bit
> >
> > Secure Renegotiation IS supported
> >
> > Compression: NONE
> >
> > Expansion: NONE
> >
> > SSL-Session:
> >
> > Protocol : TLSv1.2
> >
> > Cipher : ECDHE-RSA-AES256-GCM-SHA384
> >
> > Session-ID:
> > 5A09AF66C19A54A200221F9EFACC20642DBDCCE50099EE6836FDA0B4ECE33EF6
> >
> > Session-ID-ctx:
> >
> > Master-Key:
> > F4BD9CEA03E292AC4DC696B46E3CD1348BD954C300FAE6A07697507937B4
> > 22187B51FB0814B20CFBCAFD21A65B30BEBC
> >
> >
> > Key-Arg : None
> >
> > Krb5 Principal: None
> >
> > PSK identity: None
> >
> > PSK identity hint: None
> >
> > Start Time: 1510584166
> >
> > Timeout : 300 (sec)
> >
> > * Verify return code: 0 (ok) *
> >
> > ---
> >
> > -bash-4.1$
> >
> > --
> >
> > *Sandeep Kumar,*
> > Mobile +91-9866507368
> >
> > *“Happiness is not a destination, It is the journey”*
> >
>
--
*Sandeep Kumar,*
Mobile +91-9866507368
*“Happiness is not a destination, It is the journey”*
Re: Facing issues in enabling two way ssl in ambari for server-agent communication
Posted by Gonzalo Herreros <gh...@gmail.com>.
Did you also put the ca.crt file on the server /var/lib/ambari-server/keys?
Also in the steps you don't mention adding all the certs to the server
keystore which is in the instructions
I think it's much less error prone to let ambari create and assign the
certs it uses to communicate with agents (being careful to configure longer
life)
On 13 November 2017 at 14:54, Sandy <sa...@gmail.com> wrote:
> We are trying to setup security in server agent communication using
> instructions provided in this
>
> https://community.hortonworks.com/articles/107092/configure-
> 2-way-ssl-between-ambari-server-and-amba.html
>
> Here are the things I've done:
> 1. Obtained Certificates from CA for both server and agent machines.
> 2. Placed them in corresponding directories in both server and agent.
> 3. When we try to manually verify authentication using following command,
> two way ssl seems to be working (*openssl s_client -cert
> agent-hostname.crt -key **agent-hostname**.key
> -CAfile /var/lib/ambari-agent/keys/ca.crt -connect server-hostname:8441
> -msg) Detailed logs below*
> 4. But ambari agent throws following error
>
> INFO 2017-11-13 14:36:43,411 NetUtil.py:70 - Connecting to
> https://server-hostname:8440/connection_info
>
> INFO 2017-11-13 14:36:43,426 security.py:55 - Server require two-way SSL
> authentication. Use it instead of one-way...
>
> INFO 2017-11-13 14:36:43,426 security.py:183 - Server certicate exists, ok
>
> INFO 2017-11-13 14:36:43,426 security.py:191 - Agent key exists, ok
>
> INFO 2017-11-13 14:36:43,427 security.py:199 - Agent certificate exists, ok
>
> INFO 2017-11-13 14:36:43,427 security.py:94 - SSL Connect being called..
> connecting to the server
>
> ERROR 2017-11-13 14:36:43,432 security.py:81 - Two-way SSL authentication
> failed. Ensure that server and agent certificates were signed by the same
> CA and restart the agent.
>
> In order to receive a new agent certificate, remove existing certificate
> file from keys directory. As a workaround you can turn off two-way SSL
> authentication in server configuration(ambari.properties)
>
> Exiting..
>
> ERROR 2017-11-13 14:36:43,432 Controller.py:226 - Unable to connect to:
> https://server-hostname:8441/agent/v1/register/agent-hostname
>
> Traceback (most recent call last):
>
> File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line
> 175, in registerWithServer
>
> ret = self.sendRequest(self.registerUrl, data)
>
> File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line
> 549, in sendRequest
>
> raise IOError('Request to {0} failed due to {1}'.format(url, str
> (exception)))
>
> IOError: Request to
> https://server-hostname:8441/agent/v1/register/agent-hostname failed due
> to
> [Errno 8] _ssl.c:492: EOF occurred in violation of protocol
>
> ERROR 2017-11-13 14:36:43,433 Controller.py:227 - Error:Request to
> https://server-hostname:8441/agent/v1/register/agent-hostname failed due
> to
> [Errno 8] _ssl.c:492: EOF occurred in violation of protocol
>
> WARNING 2017-11-13 14:36:43,433 Controller.py:228 - Sleeping for 11
> seconds and then trying again
>
>
> *Can someone help ?*
>
> -bash-4.1$ *openssl s_client -cert agent-hostname.crt -key
> **agent-hostname**.key
> -CAfile /var/lib/ambari-agent/keys/ca.crt -connect server-hostname:8441
> -msg *
>
> CONNECTED(00000003)
>
> >>> TLS 1.2 Handshake [length 00f2], ClientHello
>
> ...
>
> <<< TLS 1.2 Handshake [length 0051], ServerHello
>
> ...
>
> <<< TLS 1.2 Handshake [length 0524], Certificate
>
> ...
>
> verify return:1
>
> <<< TLS 1.2 Handshake [length 0191], ServerKeyExchange
>
> ...
>
> <<< TLS 1.2 Handshake [length 00d2], CertificateRequest
>
> ...
>
> <<< TLS 1.2 Handshake [length 0004], ServerHelloDone
>
> 0e 00 00 00
>
> >>> TLS 1.2 Handshake [length 0fe2], Certificate
>
> ...
>
> >>> TLS 1.2 Handshake [length 008a], ClientKeyExchange
>
> ...
>
> >>> TLS 1.2 Handshake [length 0108], CertificateVerify
>
> ...
>
> >>> TLS 1.2 ChangeCipherSpec [length 0001]
>
> 01
>
> >>> TLS 1.2 Handshake [length 0010], Finished
>
> ---
>
> Certificate chain
>
> ---
>
> Server certificate
>
> -----BEGIN CERTIFICATE-----
>
> ...
>
> -----END CERTIFICATE-----
>
> ...
>
> ---
>
> Acceptable client certificate CA names
>
> ---
>
> SSL handshake has read 2017 bytes and written 4534 bytes
>
> ---
>
> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
>
> Server public key is 2048 bit
>
> Secure Renegotiation IS supported
>
> Compression: NONE
>
> Expansion: NONE
>
> SSL-Session:
>
> Protocol : TLSv1.2
>
> Cipher : ECDHE-RSA-AES256-GCM-SHA384
>
> Session-ID:
> 5A09AF66C19A54A200221F9EFACC20642DBDCCE50099EE6836FDA0B4ECE33EF6
>
> Session-ID-ctx:
>
> Master-Key:
> F4BD9CEA03E292AC4DC696B46E3CD1348BD954C300FAE6A07697507937B4
> 22187B51FB0814B20CFBCAFD21A65B30BEBC
>
>
> Key-Arg : None
>
> Krb5 Principal: None
>
> PSK identity: None
>
> PSK identity hint: None
>
> Start Time: 1510584166
>
> Timeout : 300 (sec)
>
> * Verify return code: 0 (ok) *
>
> ---
>
> -bash-4.1$
>
> --
>
> *Sandeep Kumar,*
> Mobile +91-9866507368
>
> *“Happiness is not a destination, It is the journey”*
>