You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Porta Léonard <le...@kudelskisecurity.com> on 2017/04/05 09:44:29 UTC
RE: Cluster instability due to java.net.SocketTimeoutException:
Read timed out
Hello,
I'm getting exactly the same issue (running NiFI 1.1.1 too)
I have tried to increase read timeout to 10 secs (jetty threads = 200) but I'm still getting the same errors.
@Alessio did you solve the issue on your side ? I am interested in the solution if you found any.
Regards,
Léonard
-----Original Message-----
From: Koji Kawamura [mailto:ijokarumawak@gmail.com]
Sent: lundi 13 février 2017 10:04
To: users@nifi.apache.org
Subject: Re: Cluster instability due to java.net.SocketTimeoutException: Read timed out
Hello,
The timeout happens when a node replicate a request to nodes to get a merged cluster wide response, so increasing nifi.cluster.node.read.timeout might help (in conf/nifi.properties).
Request replication shouldn't timeout ideally, increasing nifi.web.jetty.threads might help to increase each node's capability to handle more requests..
nifi.cluster.node.read.timeout=5 sec
nifi.web.jetty.threads=200
Would you share nifi thread dump result (./bin/nifi.sh dump, and written in logs/nifi-bootstrap.log) while you are sending request from NiFi UI and waiting its result arrives?
It might be helpful for us to find what causes the timeout or slowness.
Thanks,
Koji
On Mon, Feb 13, 2017 at 5:52 PM, Alessio Palma <al...@docomodigital.com> wrote:
> Hello again,
> this is the stack trace which is raised on the connection timeout.
>
> 2017-02-13 08:02:01,796 WARN [Replicate Request Thread-3]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request
> GET /nifi-api/flow/status to mid1-a128-3.buongiorn
> o.com:8080 due to {}
> com.sun.jersey.api.client.ClientHandlerException:
> java.net.SocketTimeoutException: Read timed out
> at
> com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(
> URLConnectionClientHandler.java:155)
> ~[jersey-client-1.19.jar:1.19]
> at com.sun.jersey.api.client.Client.handle(Client.java:652)
> ~[jersey-client-1.19.jar:1.19]
> at
> com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIP
> ContentEncodingFilter.java:123)
> ~[jersey-client-1.19.jar:1.19]
> at
> com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
> ~[jersey-client-1.19.jar:1.19]
> at
> com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
> ~[jersey-client-1.19.jar:1.19]
> at
> com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509
> )
> ~[jersey-client-1.19.jar:1.19]
> at
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolReques
> tReplicator.replicateRequest(ThreadPoolRequestReplicator.java:578)
> ~[nifi-framework-cluster-1.1.1.jar:1.1.1]
> at
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolReques
> tReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:770)
> ~[nifi-framework-cluster-1.1.1.jar:1.1.1]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511
> )
> [na:1.8.0_101]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
> ava:1142)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
> java:617)
> [na:1.8.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101] Caused
> by: java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> ~[na:1.8.0_101]
> at
> java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> ~[na:1.8.0_101]
> at java.net.SocketInputStream.read(SocketInputStream.java:170)
> ~[na:1.8.0_101]
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> ~[na:1.8.0_101]
> at
> sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
> ~[na:1.8.0_101]
> at sun.security.ssl.InputRecord.read(InputRecord.java:503)
> ~[na:1.8.0_101]
> at
> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
> ~[na:1.8.0_101]
> at
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.j
> ava:1375)
> ~[na:1.8.0_101]
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
> ~[na:1.8.0_101]
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
> ~[na:1.8.0_101]
> at
> sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:5
> 59)
> ~[na:1.8.0_101]
> at
> sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(
> AbstractDelegateHttpsURLConnection.java:185)
> ~[na:1.8.0_101]
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLCon
> nection.java:1513)
> ~[na:1.8.0_101]
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConn
> ection.java:1441)
> ~[na:1.8.0_101]
> at
> java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
> ~[na:1.8.0_101]
> at
> sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Http
> sURLConnectionImpl.java:338)
> ~[na:1.8.0_101]
> at
> com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke
> (URLConnectionClientHandler.java:253)
> ~[jersey-client-1.19.jar:1.19]
> at
> com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(
> URLConnectionClientHandler.java:153)
> ~[jersey-client-1.19.jar:1.19]
> ... 12 common frames omitted
>
> perhaps it is useful to understand the problem.
>
>
>
> ________________________________
> From: Alessio Palma <al...@docomodigital.com>
> Sent: Monday, February 13, 2017 9:31:15 AM
> To: NIFI User mailing
> Subject: Cluster instability due to java.net.SocketTimeoutException:
> Read timed out
>
> Hello all,
> sometime the cluster is unstable due to "read time out".
> Is here any parameter which I can fine tuning in order to reduce this
> problem ?
> Most timeouts (66%) are coming from
>
> com.sun.jersey.api.client.ClientHandlerException
>
> few (33%) from
>
> o.apache.nifi.controller.FlowController
>
>
> Thanks in adavance
> AP
>