You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by "QUEVILLON EMMANUEL - EXT-SAFRAN ENGINEERING SERVICES (SAFRAN)" <em...@safrangroup.com> on 2021/10/27 13:27:33 UTC
Nifi secured cluster can't send heartbeat between nodes
Hi list,
I'm facing a weird problem I can't resolve or even understand with my secured nifi cluster. Below is the situation.
We have setup a secured nifi cluster with 3 nodes, say node1,node2 and node3.
For each of theses nodes, we've manually created a SSL certificate signing request (CSR) (using a password protected private key) to be signed by our internal CA.
Once we've get the certificates signed, I've installed each node certificates following this procedure:
1) Add the full certificate chain (root + intermediate certificates) into the signed certificate.
cat nifi-nodeX.pem cert_chain.pem > full-nifi-nodeX.pem
2) Create a PKCS12 certificate using private key (.key) and full signed certificate (.pem)
openssl pkcs12 -export -in full-nifi-nodeX.pem -inkey nifi-nodeX.key -out nifi-nodeX.p12 \
-name nifi-nodeX -passin pass:"XXXXXX" -passout pass:YYYYY;
3) Import nifi-nodeX.p12 into the nifi-nodeX keystore
keytool -omportkeystore -deststorepass xxxxxx -destkeystore keystore.jks -srckeystore nifi-nodeX.p12 -srcstoretype PKCS12
4) Then added each other nifi-node certificates (.pem) into nifi-truststore
node1: add full-nifi-node2 + full-nifi-node3 into truststore
node2: add full-nifi-node1 + full-nifi-node3 into truststore
node3: add full-nifi-node2 + full-nifi-node1 into truststore
5) Restarted each node
Once each node are restarted, I can connect to the web UI, but I've got an error message saying:
For info, web UI is reachable on port 8443
Invalid State:
The Flow Controller is initializing the Data Flow.
Looking at node logs (nifi-app.log) I can see that each node cannot talk to each other and to the Coordinator to send heartbeat messages:
Nifi-node1:
INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at nifi-node2:11443; will use this address for sending heartbeat messages
INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at nifi-node2:11443. Will send Cluster Connection Request to this address
WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed unmarshalling 'CONNECTION_RESPONSE' protocol message from nifi-node2.rd1.rf1/10.108.70.39:11443 due to: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
...
Nifi-node2:
WARN [Process Cluster Protocol Request-1] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from nifi-node3 due to Empty client certificate chain
INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at nifi-node2:11443; will use this address for sending heartbeat messages
INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at nifi-node2:11443. Will send Cluster Connection Request to this address
WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: java.net.SocketException: Connection reset by peer (Write failed)
...
Nifi-node3:
INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at nifi-node2:11443; will use this address for sending heartbeat messages
INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at nifi-node2:11443. Will send Cluster Connection Request to this address
WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed unmarshalling 'CONNECTION_RESPONSE' protocol message from nifi-node2/10.108.70.39:11443 due to: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
It looks like the signed certificates are not ok regarding the logs errors.
However, trying these certificates using openssl s_client command works as expected:
openssl s_client -connect nifi-node3:11443 -cert full-nifi-node3.pem -key nifi-node3.key -pass pass:'XXXXXXX'
CONNECTED(00000003)
depth=3 C = FR, O = SAFRAN, OU = 0002 562082909, CN = SAFRAN Root CA 1
verify return:1
depth=2 C = FR, O = SAFRAN, OU = 0002 562082909, CN = SAFRAN Corporate CA 1
verify return:1
depth=1 C = FR, O = SAFRAN, OU = 0002 562082909, CN = SAFRAN Corporate Service CA 2
verify return:1
depth=0 C = FR, O = SAFRAN, OU = SAFRAN SA, OU = 0002 562082909, CN = nifi-node3
verify return:1
---
Certificate chain
0 s:/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=nifi-node3
i:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate Service CA 2
1 s:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate Service CA 2
i:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate CA 1
2 s:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate CA 1
i:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Root CA 1
3 s:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Root CA 1
i:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Root CA 1
---
Server certificate
-----BEGIN CERTIFICATE-----
....
-----END CERTIFICATE-----
subject=/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=nifi-node3
issuer=/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate Service CA 2
---
Acceptable client certificate CA names
/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=nifi-node3
/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=niif-node2
/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=nifi-node1
/C=FR/OU=SAFRAN SA/OU=0002 562082909/O=SAFRAN/CN=Safran Nifi Admin
/C=FR/OU=SAFRAN SA/OU=0002 562082909/O=SAFRAN/CN=localhost
Client Certificate Types: ECDSA sign, RSA sign, DSA sign
Requested Signature Algorithms: 0x07+0x08:0x08+0x08:ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:0x04+0x08:0x05+0x08:0x06+0x08:0x09+0x08:0x0A+0x08:0x0B+0x08:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 8911 bytes and written 8534 bytes
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 68686A816F510BED151FEBB80604862B799CD0D5DFCEA9602A9E204E9EC5741E
Session-ID-ctx:
Master-Key: CB4E24EDCAA3518494C04762965452CDC9CE993FCCAF3DBCCF76755376B808667342AF327DE5B8DE6B3B981F55B3CB90
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1635340626
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
closed
There is something I don't get!
I've tried all the above procedure without adding the full cert chain, same errors.
I've tried with autogenerated self-signed certificates using nifi-toolkit, and it works as expected, so I think there is definitely something wrong with the signed certificates but I've no clue at all what it could be.
Please could someone light my lantern, I've no more idea or way to explore.
Regards
Emmanuel
C2 - Restricted
#
" Ce courriel et les documents qui lui sont joints peuvent contenir des informations confidentielles, être soumis aux règlementations relatives au contrôle des exportations ou ayant un caractère privé. S'ils ne vous sont pas destinés, nous vous signalons qu'il est strictement interdit de les divulguer, de les reproduire ou d'en utiliser de quelque manière que ce soit le contenu. Toute exportation ou réexportation non autorisée est interdite Si ce message vous a été transmis par erreur, merci d'en informer l'expéditeur et de supprimer immédiatement de votre système informatique ce courriel ainsi que tous les documents qui y sont attachés."
******
" This e-mail and any attached documents may contain confidential or proprietary information and may be subject to export control laws and regulations. If you are not the intended recipient, you are notified that any dissemination, copying of this e-mail and any attachments thereto or use of their contents by any means whatsoever is strictly prohibited. Unauthorized export or re-export is prohibited. If you have received this e-mail in error, please advise the sender immediately and delete this e-mail and all attached documents from your computer system."
#
RE: Nifi secured cluster can't send heartbeat between nodes
Posted by "QUEVILLON EMMANUEL - EXT-SAFRAN ENGINEERING SERVICES (SAFRAN)" <em...@safrangroup.com>.
Hi Jens,
Thanks for your quick reply. After double checking certificate extended key usage, it looks they are configured as required:
openssl x509 –in full-nifi-node1.pem –noout –text
…
X509v3 Extended Key Usage:
TLS Web Client Authentication, TLS Web Server Authentication
…
Emmanuel
Emmanuel Quevillon
DevOps | Safran Analytics
• M +33 (0)6 03 44 32 86
SAFRAN Paris-Saclay
Rue des Jeunes Bois
Châteaufort
CS 80112
78772 Magny-Les-Hameaux
www.safran-group.com<http://www.safran-group.com/fr/>
[SAFRAN]
C2 - Restricted
De : Jens M. Kofoed <jm...@gmail.com>
Envoyé : mercredi 27 octobre 2021 15:40
À : users@nifi.apache.org
Objet : Re: Nifi secured cluster can't send heartbeat between nodes
Hi
I don’t know if this will help. But the certificate used by nifi needs be both a server auth and client auth. Normally certificates are only one of them. The nifi certificate use the server auth for the web ui and when other servers connect to it. It is using the client auth when nifi talks to other nifi servers.
Regards
Jens
Den 27. okt. 2021 kl. 15.27 skrev QUEVILLON EMMANUEL - EXT-SAFRAN ENGINEERING SERVICES (SAFRAN) <em...@safrangroup.com>>:
Hi list,
I’m facing a weird problem I can’t resolve or even understand with my secured nifi cluster. Below is the situation.
We have setup a secured nifi cluster with 3 nodes, say node1,node2 and node3.
For each of theses nodes, we’ve manually created a SSL certificate signing request (CSR) (using a password protected private key) to be signed by our internal CA.
Once we’ve get the certificates signed, I’ve installed each node certificates following this procedure:
1) Add the full certificate chain (root + intermediate certificates) into the signed certificate.
cat nifi-nodeX.pem cert_chain.pem > full-nifi-nodeX.pem
2) Create a PKCS12 certificate using private key (.key) and full signed certificate (.pem)
openssl pkcs12 -export -in full-nifi-nodeX.pem -inkey nifi-nodeX.key -out nifi-nodeX.p12 \
-name nifi-nodeX -passin pass:"XXXXXX" -passout pass:YYYYY;
3) Import nifi-nodeX.p12 into the nifi-nodeX keystore
keytool –omportkeystore –deststorepass xxxxxx –destkeystore keystore.jks –srckeystore nifi-nodeX.p12 –srcstoretype PKCS12
4) Then added each other nifi-node certificates (.pem) into nifi-truststore
node1: add full-nifi-node2 + full-nifi-node3 into truststore
node2: add full-nifi-node1 + full-nifi-node3 into truststore
node3: add full-nifi-node2 + full-nifi-node1 into truststore
5) Restarted each node
Once each node are restarted, I can connect to the web UI, but I’ve got an error message saying:
For info, web UI is reachable on port 8443
Invalid State:
The Flow Controller is initializing the Data Flow.
Looking at node logs (nifi-app.log) I can see that each node cannot talk to each other and to the Coordinator to send heartbeat messages:
Nifi-node1:
INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at nifi-node2:11443; will use this address for sending heartbeat messages
INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at nifi-node2:11443. Will send Cluster Connection Request to this address
WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed unmarshalling 'CONNECTION_RESPONSE' protocol message from nifi-node2.rd1.rf1/10.108.70.39:11443 due to: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
…
Nifi-node2:
WARN [Process Cluster Protocol Request-1] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from nifi-node3 due to Empty client certificate chain
INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at nifi-node2:11443; will use this address for sending heartbeat messages
INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at nifi-node2:11443. Will send Cluster Connection Request to this address
WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: java.net.SocketException: Connection reset by peer (Write failed)
…
Nifi-node3:
INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at nifi-node2:11443; will use this address for sending heartbeat messages
INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at nifi-node2:11443. Will send Cluster Connection Request to this address
WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed unmarshalling 'CONNECTION_RESPONSE' protocol message from nifi-node2/10.108.70.39:11443 due to: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
It looks like the signed certificates are not ok regarding the logs errors.
However, trying these certificates using openssl s_client command works as expected:
openssl s_client -connect nifi-node3:11443 -cert full-nifi-node3.pem -key nifi-node3.key -pass pass:'XXXXXXX’
CONNECTED(00000003)
depth=3 C = FR, O = SAFRAN, OU = 0002 562082909, CN = SAFRAN Root CA 1
verify return:1
depth=2 C = FR, O = SAFRAN, OU = 0002 562082909, CN = SAFRAN Corporate CA 1
verify return:1
depth=1 C = FR, O = SAFRAN, OU = 0002 562082909, CN = SAFRAN Corporate Service CA 2
verify return:1
depth=0 C = FR, O = SAFRAN, OU = SAFRAN SA, OU = 0002 562082909, CN = nifi-node3
verify return:1
---
Certificate chain
0 s:/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=nifi-node3
i:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate Service CA 2
1 s:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate Service CA 2
i:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate CA 1
2 s:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate CA 1
i:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Root CA 1
3 s:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Root CA 1
i:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Root CA 1
---
Server certificate
-----BEGIN CERTIFICATE-----
….
-----END CERTIFICATE-----
subject=/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=nifi-node3
issuer=/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate Service CA 2
---
Acceptable client certificate CA names
/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=nifi-node3
/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=niif-node2
/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=nifi-node1
/C=FR/OU=SAFRAN SA/OU=0002 562082909/O=SAFRAN/CN=Safran Nifi Admin
/C=FR/OU=SAFRAN SA/OU=0002 562082909/O=SAFRAN/CN=localhost
Client Certificate Types: ECDSA sign, RSA sign, DSA sign
Requested Signature Algorithms: 0x07+0x08:0x08+0x08:ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:0x04+0x08:0x05+0x08:0x06+0x08:0x09+0x08:0x0A+0x08:0x0B+0x08:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 8911 bytes and written 8534 bytes
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 68686A816F510BED151FEBB80604862B799CD0D5DFCEA9602A9E204E9EC5741E
Session-ID-ctx:
Master-Key: CB4E24EDCAA3518494C04762965452CDC9CE993FCCAF3DBCCF76755376B808667342AF327DE5B8DE6B3B981F55B3CB90
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1635340626
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
closed
There is something I don’t get!
I’ve tried all the above procedure without adding the full cert chain, same errors.
I’ve tried with autogenerated self-signed certificates using nifi-toolkit, and it works as expected, so I think there is definitely something wrong with the signed certificates but I’ve no clue at all what it could be.
Please could someone light my lantern, I’ve no more idea or way to explore.
Regards
Emmanuel
C2 - Restricted
#
" Ce courriel et les documents qui lui sont joints peuvent contenir des informations confidentielles, être soumis aux règlementations relatives au contrôle des exportations ou ayant un caractère privé. S'ils ne vous sont pas destinés, nous vous signalons qu'il est strictement interdit de les divulguer, de les reproduire ou d'en utiliser de quelque manière que ce soit le contenu. Toute exportation ou réexportation non autorisée est interdite Si ce message vous a été transmis par erreur, merci d'en informer l'expéditeur et de supprimer immédiatement de votre système informatique ce courriel ainsi que tous les documents qui y sont attachés."
******
" This e-mail and any attached documents may contain confidential or proprietary information and may be subject to export control laws and regulations. If you are not the intended recipient, you are notified that any dissemination, copying of this e-mail and any attachments thereto or use of their contents by any means whatsoever is strictly prohibited. Unauthorized export or re-export is prohibited. If you have received this e-mail in error, please advise the sender immediately and delete this e-mail and all attached documents from your computer system."
#
#
" Ce courriel et les documents qui lui sont joints peuvent contenir des informations confidentielles, être soumis aux règlementations relatives au contrôle des exportations ou ayant un caractère privé. S'ils ne vous sont pas destinés, nous vous signalons qu'il est strictement interdit de les divulguer, de les reproduire ou d'en utiliser de quelque manière que ce soit le contenu. Toute exportation ou réexportation non autorisée est interdite Si ce message vous a été transmis par erreur, merci d'en informer l'expéditeur et de supprimer immédiatement de votre système informatique ce courriel ainsi que tous les documents qui y sont attachés."
******
" This e-mail and any attached documents may contain confidential or proprietary information and may be subject to export control laws and regulations. If you are not the intended recipient, you are notified that any dissemination, copying of this e-mail and any attachments thereto or use of their contents by any means whatsoever is strictly prohibited. Unauthorized export or re-export is prohibited. If you have received this e-mail in error, please advise the sender immediately and delete this e-mail and all attached documents from your computer system."
#
Re: Nifi secured cluster can't send heartbeat between nodes
Posted by "Jens M. Kofoed" <jm...@gmail.com>.
Hi
I don’t know if this will help. But the certificate used by nifi needs be both a server auth and client auth. Normally certificates are only one of them. The nifi certificate use the server auth for the web ui and when other servers connect to it. It is using the client auth when nifi talks to other nifi servers.
Regards
Jens
> Den 27. okt. 2021 kl. 15.27 skrev QUEVILLON EMMANUEL - EXT-SAFRAN ENGINEERING SERVICES (SAFRAN) <em...@safrangroup.com>:
>
> Hi list,
>
> I’m facing a weird problem I can’t resolve or even understand with my secured nifi cluster. Below is the situation.
> We have setup a secured nifi cluster with 3 nodes, say node1,node2 and node3.
> For each of theses nodes, we’ve manually created a SSL certificate signing request (CSR) (using a password protected private key) to be signed by our internal CA.
> Once we’ve get the certificates signed, I’ve installed each node certificates following this procedure:
>
> 1) Add the full certificate chain (root + intermediate certificates) into the signed certificate.
> cat nifi-nodeX.pem cert_chain.pem > full-nifi-nodeX.pem
> 2) Create a PKCS12 certificate using private key (.key) and full signed certificate (.pem)
> openssl pkcs12 -export -in full-nifi-nodeX.pem -inkey nifi-nodeX.key -out nifi-nodeX.p12 \
> -name nifi-nodeX -passin pass:"XXXXXX" -passout pass:YYYYY;
> 3) Import nifi-nodeX.p12 into the nifi-nodeX keystore
> keytool –omportkeystore –deststorepass xxxxxx –destkeystore keystore.jks –srckeystore nifi-nodeX.p12 –srcstoretype PKCS12
> 4) Then added each other nifi-node certificates (.pem) into nifi-truststore
> node1: add full-nifi-node2 + full-nifi-node3 into truststore
> node2: add full-nifi-node1 + full-nifi-node3 into truststore
> node3: add full-nifi-node2 + full-nifi-node1 into truststore
> 5) Restarted each node
>
> Once each node are restarted, I can connect to the web UI, but I’ve got an error message saying:
>
> For info, web UI is reachable on port 8443
>
> Invalid State:
> The Flow Controller is initializing the Data Flow.
>
> Looking at node logs (nifi-app.log) I can see that each node cannot talk to each other and to the Coordinator to send heartbeat messages:
>
> Nifi-node1:
>
> INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at nifi-node2:11443; will use this address for sending heartbeat messages
> INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at nifi-node2:11443. Will send Cluster Connection Request to this address
> WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed unmarshalling 'CONNECTION_RESPONSE' protocol message from nifi-node2.rd1.rf1/10.108.70.39:11443 due to: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
> …
>
> Nifi-node2:
>
> WARN [Process Cluster Protocol Request-1] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from nifi-node3 due to Empty client certificate chain
> INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at nifi-node2:11443; will use this address for sending heartbeat messages
> INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at nifi-node2:11443. Will send Cluster Connection Request to this address
> WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: java.net.SocketException: Connection reset by peer (Write failed)
> …
>
> Nifi-node3:
>
> INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at nifi-node2:11443; will use this address for sending heartbeat messages
> INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at nifi-node2:11443. Will send Cluster Connection Request to this address
> WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed unmarshalling 'CONNECTION_RESPONSE' protocol message from nifi-node2/10.108.70.39:11443 due to: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
>
> It looks like the signed certificates are not ok regarding the logs errors.
> However, trying these certificates using openssl s_client command works as expected:
>
> openssl s_client -connect nifi-node3:11443 -cert full-nifi-node3.pem -key nifi-node3.key -pass pass:'XXXXXXX’
> CONNECTED(00000003)
> depth=3 C = FR, O = SAFRAN, OU = 0002 562082909, CN = SAFRAN Root CA 1
> verify return:1
> depth=2 C = FR, O = SAFRAN, OU = 0002 562082909, CN = SAFRAN Corporate CA 1
> verify return:1
> depth=1 C = FR, O = SAFRAN, OU = 0002 562082909, CN = SAFRAN Corporate Service CA 2
> verify return:1
> depth=0 C = FR, O = SAFRAN, OU = SAFRAN SA, OU = 0002 562082909, CN = nifi-node3
> verify return:1
> ---
> Certificate chain
> 0 s:/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=nifi-node3
> i:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate Service CA 2
> 1 s:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate Service CA 2
> i:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate CA 1
> 2 s:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate CA 1
> i:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Root CA 1
> 3 s:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Root CA 1
> i:/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Root CA 1
> ---
> Server certificate
> -----BEGIN CERTIFICATE-----
> ….
> -----END CERTIFICATE-----
> subject=/C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=nifi-node3
> issuer=/C=FR/O=SAFRAN/OU=0002 562082909/CN=SAFRAN Corporate Service CA 2
> ---
> Acceptable client certificate CA names
> /C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=nifi-node3
> /C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=niif-node2
> /C=FR/O=SAFRAN/OU=SAFRAN SA/OU=0002 562082909/CN=nifi-node1
> /C=FR/OU=SAFRAN SA/OU=0002 562082909/O=SAFRAN/CN=Safran Nifi Admin
> /C=FR/OU=SAFRAN SA/OU=0002 562082909/O=SAFRAN/CN=localhost
> Client Certificate Types: ECDSA sign, RSA sign, DSA sign
> Requested Signature Algorithms: 0x07+0x08:0x08+0x08:ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:0x04+0x08:0x05+0x08:0x06+0x08:0x09+0x08:0x0A+0x08:0x0B+0x08:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1
> Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1
> Peer signing digest: SHA512
> Server Temp Key: ECDH, P-256, 256 bits
> ---
> SSL handshake has read 8911 bytes and written 8534 bytes
> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
> Server public key is 4096 bit
> Secure Renegotiation IS supported
> Compression: NONE
> Expansion: NONE
> No ALPN negotiated
> SSL-Session:
> Protocol : TLSv1.2
> Cipher : ECDHE-RSA-AES256-GCM-SHA384
> Session-ID: 68686A816F510BED151FEBB80604862B799CD0D5DFCEA9602A9E204E9EC5741E
> Session-ID-ctx:
> Master-Key: CB4E24EDCAA3518494C04762965452CDC9CE993FCCAF3DBCCF76755376B808667342AF327DE5B8DE6B3B981F55B3CB90
> Key-Arg : None
> Krb5 Principal: None
> PSK identity: None
> PSK identity hint: None
> Start Time: 1635340626
> Timeout : 300 (sec)
> Verify return code: 0 (ok)
> ---
>
> closed
>
> There is something I don’t get!
> I’ve tried all the above procedure without adding the full cert chain, same errors.
> I’ve tried with autogenerated self-signed certificates using nifi-toolkit, and it works as expected, so I think there is definitely something wrong with the signed certificates but I’ve no clue at all what it could be.
>
> Please could someone light my lantern, I’ve no more idea or way to explore.
> Regards
>
> Emmanuel
>
> C2 - Restricted
>
> #
> " Ce courriel et les documents qui lui sont joints peuvent contenir des informations confidentielles, être soumis aux règlementations relatives au contrôle des exportations ou ayant un caractère privé. S'ils ne vous sont pas destinés, nous vous signalons qu'il est strictement interdit de les divulguer, de les reproduire ou d'en utiliser de quelque manière que ce soit le contenu. Toute exportation ou réexportation non autorisée est interdite Si ce message vous a été transmis par erreur, merci d'en informer l'expéditeur et de supprimer immédiatement de votre système informatique ce courriel ainsi que tous les documents qui y sont attachés."
> ******
> " This e-mail and any attached documents may contain confidential or proprietary information and may be subject to export control laws and regulations. If you are not the intended recipient, you are notified that any dissemination, copying of this e-mail and any attachments thereto or use of their contents by any means whatsoever is strictly prohibited. Unauthorized export or re-export is prohibited. If you have received this e-mail in error, please advise the sender immediately and delete this e-mail and all attached documents from your computer system."
> #