You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Koji Kawamura <ij...@gmail.com> on 2019/03/05 00:36:07 UTC
Re: SSLHandshake Exception from Site-to-Site
Hi Nadeem,
How many S2S clients are connecting to your NiFi? And how many NiFi
nodes does your remote NiFi have?
I've encountered the same error message when I conducted a test using
hundreds of S2S client connecting to a single NiFi node.
It happened in a situation like followings:
1. A S2S client connects to the NiFi node
2. The NiFi node accepts the connection, spawns new thread to process
further communication [Site-to-Site Worker Thread-N]
3. But the NiFi node is not able to process incoming connections fast
enough, and when the node starts SSL hand-shake process, the client
has already disconnected.
In my case, setting longer timeout at S2S clients helped accepting
more concurrent connections. But also this can be an indication
suggesting the need of having more nodes (if the message is logged
from the similar situation with mine).
Another possibility is as the message says, a malicious user is
actually sending SSL truncation attack.
Thanks,
Koji
On Fri, Mar 1, 2019 at 1:19 AM Mohammed Nadeem <na...@gmail.com> wrote:
>
> Hi,
>
> Can someone please help me resolving SSLHandshake issue (Site-to-Site) which
> I'm getting in logs. This ERROR doesn't impact us from accessing the NiFi
> canvas or any calls we make from Nifi components (like SSL Context Service).
> This is something which keeps on throwing every now and then in
> nifi-app.logs
>
> Below, is the error we get in the logs
>
> ERROR [Site-to-Site Worker Thread-138]
> o.a.n.r.io.socket.ssl.SSLSocketChannel
> org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel@938965a Failed to
> connect due to {}
> javax.net.ssl.SSLHandshakeException: Reached End-of-File marker while
> performing handshake
> at
> org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel.performHandshake(SSLSocketChannel.java:248)
> at
> org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel.connect(SSLSocketChannel.java:163)
> at
> org.apache.nifi.remote.SocketRemoteSiteListener$1$1.run(SocketRemoteSiteListener.java:168)
> at java.lang.Thread.run(Thread.java:748)
>
> ERROR [Site-to-Site Worker Thread-138]
> o.a.nifi.remote.SocketRemoteSiteListener RemoteSiteListener Unable to accept
> connection from Socket[unconnected] due to javax.net.ssl.SSLException:
> Inbound closed before receiving peer's close_notify: possible truncation
> attack?
>
> Setup,
> CA Server is running on separate host ( eg, ca_server_host ) which generates
> self-signed certificates
> Each Nifi instance calls CA to get the keystore, trustore etc like the
> necessary certs
>
> Please help me understand the issue, I have gone through many resources
> online but I wasn't able to resolve,
>
> Thanks,
> Nadeem
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: SSLHandshake Exception from Site-to-Site
Posted by Mohammed Nadeem <na...@gmail.com>.
Thanks again Koji for replying and understanding my concern,
I did apply the changes you suggested but still i'm getting same
SSLHandshake error. I believe the Site2Site Remote Listener doesn't run a
server socket with the hostname we specify in *'nifi.remote.input.host'* in
nifi.properties instead it uses wildcard ipaddress *0.0.0.0* to bind
serversocket to all network interfaces to listen for incoming request from
site-to-site client, please check this line of code - link for code line in
socketremotelisterner class
<https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-site-to-site/src/main/java/org/apache/nifi/remote/SocketRemoteSiteListener.java#L97>
. Since the serversocket is listening on all network interfaces and it waits
for the client to accept the connection, it runs a continuous while loop
waiting to accept the connection from client, please check this segment of
code as well
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-site-to-site/src/main/java/org/apache/nifi/remote/SocketRemoteSiteListener.java#L123-L129
<https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-site-to-site/src/main/java/org/apache/nifi/remote/SocketRemoteSiteListener.java#L123-L129>
. In kubernetes, some anonymous client is able to get through the connection
to serversocket and while performing handshake its throwing the SSLHandshake
error.
This anonymous client has hostname which as ip-10-200-25-3.compute.internal
with randorm port and thats how the socketremotelisterner tries to creates
SSLsocketchannel with this anonymous client and it throws the error.
Please see the attached image i created explaining the root cause for this
error, Please do suggest if I'm correct or wrong.
<http://apache-nifi-developer-list.39713.n7.nabble.com/file/t869/s2s-error.png>
Thanks again,
Nadeem
--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: SSLHandshake Exception from Site-to-Site
Posted by Koji Kawamura <ij...@gmail.com>.
Hi Nadeem,
> nifi.remote.input.host=<hostname_of_the_self_node_container>
This property is used for how S2S server introduces itself to S2S
clients for further network communication.
For example, let's say if the server has 2 ip addresses, private and
public, and the public ip is bounded to a fqdn. hostnames for the
server would be:
- private: ip-10-200-46-112.us-west-2.compute.internal
- public: nifi1.example.com
In that case, the property should be set as
nifi.remote.input.host=nifi1.example.com
I don't have much experience with Kubernetes (K8s), but usually some
naming and port mapping configurations should be used in order to
expose such public API endpoint in an environment like K8s or
containers.
Or use service name to make communication between containers successful.
https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/
Furthermore, if you need to expose your NiFi S2S running on K8s so
that S2S client running outside of the K8s cluster can communicate,
then I think you will need to deploy a load-balancer.
https://kubernetes.io/docs/tutorials/stateless-application/expose-external-ip-address/
> [Site-to-Site Worker Thread-235] o.a.nifi.remote.SocketRemoteSiteListener
> org.apache.nifi.remote.SocketRemoteSiteListener$1$1@74dd1923 Connection URL
> is nifi://ip-10-200-46-112.us-west-2.compute.internal:22343*
This log is written by S2S worker thread, that is working at the S2S
server accepting incoming connections.
The Connection URL here represents the associated peer, which is a S2S
client connecting to the server.
That's why it shows internal hostname and random port.
Thanks,
Koji
On Thu, Mar 7, 2019 at 12:14 AM Mohammed Nadeem <na...@gmail.com> wrote:
>
> Thank you so much Koji for replying,
>
> This issue of SSL Handshake we see is for a single node cluster instance,
> where our NiFi application has been deployed in Kubernetes container, Here
> is the below configuration we did for site-to-site in nifi.properties file
> for a single cluster node.
>
> # Site to Site properties
> nifi.remote.input.host=<hostname_of_the_self_node_container>
> nifi.remote.input.secure=true
> nifi.remote.input.socket.port=9443
> nifi.remote.input.http.enabled=false
>
> I was trying to understand how site-to-site works internally by going
> through source code and also debugging parallel to how it does
> communication. I found couple of observations from my analysis
>
> 1. First off, I believe when you give same hostname as nifi application
> running in a container for site-to-site in nifi.properties for single
> cluster node, the internal site-to-site java code doesn't get the hostname
> of the self node when asked for cluster nodes information (NodeInFormant),
> instead it gives some other private ip hostname. In the logs we see -* DEBUG
> [Site-to-Site Worker Thread-235] o.a.nifi.remote.SocketRemoteSiteListener
> org.apache.nifi.remote.SocketRemoteSiteListener$1$1@74dd1923 Connection URL
> is nifi://ip-10-200-46-112.us-west-2.compute.internal:22343*
> From above debug log, I see the internal java code is not recognizing that
> its a docker container instead its trying to connect with unknown hostname
> name with random port. I believe due to incapable of recognizing its a
> container instead returning some kubernetes node ip address, its throwing
> the ssl handshake error
>
> Interesting thing is, When the port 'nifi.remote.input.socket.port' (9443)
> was reachable at a container level, we see the above SSLHandshake error with
> site-to-site worker thread trying to hit different hostname from the point-1
> I mentioned above, when we blocked this port from the container, the
> SSLHandshake error went away, we no longer see when blocking the port from
> 'nifi.remote.input.socket.port' in properties. I'm not sure if this make
> sense but I want to understand how site-to-site works internally in detail.
>
> If above of my observations are incorrect or something needs to be done,
> please help me in understanding. Please Bryan, Pierre, Marks, Koji or any
> NiFi experts pleas help me understanding this. I have gone through almost
> all blogs and etc.
>
> Please suggest the solution,
>
> Thanks,
> Nadeem
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: SSLHandshake Exception from Site-to-Site
Posted by Mohammed Nadeem <na...@gmail.com>.
Thank you so much Koji for replying,
This issue of SSL Handshake we see is for a single node cluster instance,
where our NiFi application has been deployed in Kubernetes container, Here
is the below configuration we did for site-to-site in nifi.properties file
for a single cluster node.
# Site to Site properties
nifi.remote.input.host=<hostname_of_the_self_node_container>
nifi.remote.input.secure=true
nifi.remote.input.socket.port=9443
nifi.remote.input.http.enabled=false
I was trying to understand how site-to-site works internally by going
through source code and also debugging parallel to how it does
communication. I found couple of observations from my analysis
1. First off, I believe when you give same hostname as nifi application
running in a container for site-to-site in nifi.properties for single
cluster node, the internal site-to-site java code doesn't get the hostname
of the self node when asked for cluster nodes information (NodeInFormant),
instead it gives some other private ip hostname. In the logs we see -* DEBUG
[Site-to-Site Worker Thread-235] o.a.nifi.remote.SocketRemoteSiteListener
org.apache.nifi.remote.SocketRemoteSiteListener$1$1@74dd1923 Connection URL
is nifi://ip-10-200-46-112.us-west-2.compute.internal:22343*
From above debug log, I see the internal java code is not recognizing that
its a docker container instead its trying to connect with unknown hostname
name with random port. I believe due to incapable of recognizing its a
container instead returning some kubernetes node ip address, its throwing
the ssl handshake error
Interesting thing is, When the port 'nifi.remote.input.socket.port' (9443)
was reachable at a container level, we see the above SSLHandshake error with
site-to-site worker thread trying to hit different hostname from the point-1
I mentioned above, when we blocked this port from the container, the
SSLHandshake error went away, we no longer see when blocking the port from
'nifi.remote.input.socket.port' in properties. I'm not sure if this make
sense but I want to understand how site-to-site works internally in detail.
If above of my observations are incorrect or something needs to be done,
please help me in understanding. Please Bryan, Pierre, Marks, Koji or any
NiFi experts pleas help me understanding this. I have gone through almost
all blogs and etc.
Please suggest the solution,
Thanks,
Nadeem
--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/