You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Alessio Palma <al...@docomodigital.com> on 2017/02/13 08:31:15 UTC
Cluster instability due to java.net.SocketTimeoutException: Read
timed out
Hello all,
sometime the cluster is unstable due to "read time out".
Is here any parameter which I can fine tuning in order to reduce this problem ?
Most timeouts (66%) are coming from
com.sun.jersey.api.client.ClientHandlerException
few (33%) from
o.apache.nifi.controller.FlowController
Thanks in adavance
AP
RE: Cluster instability due to java.net.SocketTimeoutException:
Read timed out
Posted by Porta Léonard <le...@kudelskisecurity.com>.
Hello,
I'm getting exactly the same issue (running NiFI 1.1.1 too)
I have tried to increase read timeout to 10 secs (jetty threads = 200) but I'm still getting the same errors.
@Alessio did you solve the issue on your side ? I am interested in the solution if you found any.
Regards,
Léonard
-----Original Message-----
From: Koji Kawamura [mailto:ijokarumawak@gmail.com]
Sent: lundi 13 février 2017 10:04
To: users@nifi.apache.org
Subject: Re: Cluster instability due to java.net.SocketTimeoutException: Read timed out
Hello,
The timeout happens when a node replicate a request to nodes to get a merged cluster wide response, so increasing nifi.cluster.node.read.timeout might help (in conf/nifi.properties).
Request replication shouldn't timeout ideally, increasing nifi.web.jetty.threads might help to increase each node's capability to handle more requests..
nifi.cluster.node.read.timeout=5 sec
nifi.web.jetty.threads=200
Would you share nifi thread dump result (./bin/nifi.sh dump, and written in logs/nifi-bootstrap.log) while you are sending request from NiFi UI and waiting its result arrives?
It might be helpful for us to find what causes the timeout or slowness.
Thanks,
Koji
On Mon, Feb 13, 2017 at 5:52 PM, Alessio Palma <al...@docomodigital.com> wrote:
> Hello again,
> this is the stack trace which is raised on the connection timeout.
>
> 2017-02-13 08:02:01,796 WARN [Replicate Request Thread-3]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request
> GET /nifi-api/flow/status to mid1-a128-3.buongiorn
> o.com:8080 due to {}
> com.sun.jersey.api.client.ClientHandlerException:
> java.net.SocketTimeoutException: Read timed out
> at
> com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(
> URLConnectionClientHandler.java:155)
> ~[jersey-client-1.19.jar:1.19]
> at com.sun.jersey.api.client.Client.handle(Client.java:652)
> ~[jersey-client-1.19.jar:1.19]
> at
> com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIP
> ContentEncodingFilter.java:123)
> ~[jersey-client-1.19.jar:1.19]
> at
> com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
> ~[jersey-client-1.19.jar:1.19]
> at
> com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
> ~[jersey-client-1.19.jar:1.19]
> at
> com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509
> )
> ~[jersey-client-1.19.jar:1.19]
> at
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolReques
> tReplicator.replicateRequest(ThreadPoolRequestReplicator.java:578)
> ~[nifi-framework-cluster-1.1.1.jar:1.1.1]
> at
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolReques
> tReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:770)
> ~[nifi-framework-cluster-1.1.1.jar:1.1.1]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511
> )
> [na:1.8.0_101]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
> ava:1142)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
> java:617)
> [na:1.8.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101] Caused
> by: java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> ~[na:1.8.0_101]
> at
> java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> ~[na:1.8.0_101]
> at java.net.SocketInputStream.read(SocketInputStream.java:170)
> ~[na:1.8.0_101]
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> ~[na:1.8.0_101]
> at
> sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
> ~[na:1.8.0_101]
> at sun.security.ssl.InputRecord.read(InputRecord.java:503)
> ~[na:1.8.0_101]
> at
> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
> ~[na:1.8.0_101]
> at
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.j
> ava:1375)
> ~[na:1.8.0_101]
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
> ~[na:1.8.0_101]
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
> ~[na:1.8.0_101]
> at
> sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:5
> 59)
> ~[na:1.8.0_101]
> at
> sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(
> AbstractDelegateHttpsURLConnection.java:185)
> ~[na:1.8.0_101]
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLCon
> nection.java:1513)
> ~[na:1.8.0_101]
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConn
> ection.java:1441)
> ~[na:1.8.0_101]
> at
> java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
> ~[na:1.8.0_101]
> at
> sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Http
> sURLConnectionImpl.java:338)
> ~[na:1.8.0_101]
> at
> com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke
> (URLConnectionClientHandler.java:253)
> ~[jersey-client-1.19.jar:1.19]
> at
> com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(
> URLConnectionClientHandler.java:153)
> ~[jersey-client-1.19.jar:1.19]
> ... 12 common frames omitted
>
> perhaps it is useful to understand the problem.
>
>
>
> ________________________________
> From: Alessio Palma <al...@docomodigital.com>
> Sent: Monday, February 13, 2017 9:31:15 AM
> To: NIFI User mailing
> Subject: Cluster instability due to java.net.SocketTimeoutException:
> Read timed out
>
> Hello all,
> sometime the cluster is unstable due to "read time out".
> Is here any parameter which I can fine tuning in order to reduce this
> problem ?
> Most timeouts (66%) are coming from
>
> com.sun.jersey.api.client.ClientHandlerException
>
> few (33%) from
>
> o.apache.nifi.controller.FlowController
>
>
> Thanks in adavance
> AP
>
Re: Cluster instability due to java.net.SocketTimeoutException: Read
timed out
Posted by Koji Kawamura <ij...@gmail.com>.
Hello,
The timeout happens when a node replicate a request to nodes to get a
merged cluster wide response, so increasing
nifi.cluster.node.read.timeout might help (in conf/nifi.properties).
Request replication shouldn't timeout ideally, increasing
nifi.web.jetty.threads might help to increase each node's capability
to handle more requests..
nifi.cluster.node.read.timeout=5 sec
nifi.web.jetty.threads=200
Would you share nifi thread dump result (./bin/nifi.sh dump, and
written in logs/nifi-bootstrap.log) while you are sending request from
NiFi UI and waiting its result arrives?
It might be helpful for us to find what causes the timeout or slowness.
Thanks,
Koji
On Mon, Feb 13, 2017 at 5:52 PM, Alessio Palma
<al...@docomodigital.com> wrote:
> Hello again,
> this is the stack trace which is raised on the connection timeout.
>
> 2017-02-13 08:02:01,796 WARN [Replicate Request Thread-3]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET
> /nifi-api/flow/status to mid1-a128-3.buongiorn
> o.com:8080 due to {}
> com.sun.jersey.api.client.ClientHandlerException:
> java.net.SocketTimeoutException: Read timed out
> at
> com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
> ~[jersey-client-1.19.jar:1.19]
> at com.sun.jersey.api.client.Client.handle(Client.java:652)
> ~[jersey-client-1.19.jar:1.19]
> at
> com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:123)
> ~[jersey-client-1.19.jar:1.19]
> at
> com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
> ~[jersey-client-1.19.jar:1.19]
> at
> com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
> ~[jersey-client-1.19.jar:1.19]
> at
> com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509)
> ~[jersey-client-1.19.jar:1.19]
> at
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:578)
> ~[nifi-framework-cluster-1.1.1.jar:1.1.1]
> at
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:770)
> ~[nifi-framework-cluster-1.1.1.jar:1.1.1]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_101]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
> Caused by: java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> ~[na:1.8.0_101]
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> ~[na:1.8.0_101]
> at java.net.SocketInputStream.read(SocketInputStream.java:170)
> ~[na:1.8.0_101]
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> ~[na:1.8.0_101]
> at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
> ~[na:1.8.0_101]
> at sun.security.ssl.InputRecord.read(InputRecord.java:503)
> ~[na:1.8.0_101]
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
> ~[na:1.8.0_101]
> at
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
> ~[na:1.8.0_101]
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
> ~[na:1.8.0_101]
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
> ~[na:1.8.0_101]
> at
> sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
> ~[na:1.8.0_101]
> at
> sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
> ~[na:1.8.0_101]
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1513)
> ~[na:1.8.0_101]
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
> ~[na:1.8.0_101]
> at
> java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
> ~[na:1.8.0_101]
> at
> sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
> ~[na:1.8.0_101]
> at
> com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253)
> ~[jersey-client-1.19.jar:1.19]
> at
> com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
> ~[jersey-client-1.19.jar:1.19]
> ... 12 common frames omitted
>
> perhaps it is useful to understand the problem.
>
>
>
> ________________________________
> From: Alessio Palma <al...@docomodigital.com>
> Sent: Monday, February 13, 2017 9:31:15 AM
> To: NIFI User mailing
> Subject: Cluster instability due to java.net.SocketTimeoutException: Read
> timed out
>
> Hello all,
> sometime the cluster is unstable due to "read time out".
> Is here any parameter which I can fine tuning in order to reduce this
> problem ?
> Most timeouts (66%) are coming from
>
> com.sun.jersey.api.client.ClientHandlerException
>
> few (33%) from
>
> o.apache.nifi.controller.FlowController
>
>
> Thanks in adavance
> AP
>
Re: Cluster instability due to java.net.SocketTimeoutException: Read
timed out
Posted by Alessio Palma <al...@docomodigital.com>.
Hello again,
this is the stack trace which is raised on the connection timeout.
2017-02-13 08:02:01,796 WARN [Replicate Request Thread-3] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET /nifi-api/flow/status to mid1-a128-3.buongiorn
o.com:8080 due to {}
com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.Client.handle(Client.java:652) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:123) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509) ~[jersey-client-1.19.jar:1.19]
at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:578) ~[nifi-framework-cluster-1.1.1.jar:1.1.1]
at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:770) ~[nifi-framework-cluster-1.1.1.jar:1.1.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_101]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.8.0_101]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[na:1.8.0_101]
at java.net.SocketInputStream.read(SocketInputStream.java:170) ~[na:1.8.0_101]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[na:1.8.0_101]
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) ~[na:1.8.0_101]
at sun.security.ssl.InputRecord.read(InputRecord.java:503) ~[na:1.8.0_101]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) ~[na:1.8.0_101]
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) ~[na:1.8.0_101]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403) ~[na:1.8.0_101]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387) ~[na:1.8.0_101]
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) ~[na:1.8.0_101]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) ~[na:1.8.0_101]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1513) ~[na:1.8.0_101]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441) ~[na:1.8.0_101]
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) ~[na:1.8.0_101]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338) ~[na:1.8.0_101]
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153) ~[jersey-client-1.19.jar:1.19]
... 12 common frames omitted
perhaps it is useful to understand the problem.
________________________________
From: Alessio Palma <al...@docomodigital.com>
Sent: Monday, February 13, 2017 9:31:15 AM
To: NIFI User mailing
Subject: Cluster instability due to java.net.SocketTimeoutException: Read timed out
Hello all,
sometime the cluster is unstable due to "read time out".
Is here any parameter which I can fine tuning in order to reduce this problem ?
Most timeouts (66%) are coming from
com.sun.jersey.api.client.ClientHandlerException
few (33%) from
o.apache.nifi.controller.FlowController
Thanks in adavance
AP