[jira] [Created] (NIFI-3317) Under load, ThreadPoolRequestReplicator is experiencing SocketTimeoutExceptions

["Jeff Storck (JIRA)" <ji...@apache.org>]

Jeff Storck created NIFI-3317:
---------------------------------

             Summary: Under load, ThreadPoolRequestReplicator is experiencing SocketTimeoutExceptions
                 Key: NIFI-3317
                 URL: https://issues.apache.org/jira/browse/NIFI-3317
             Project: Apache NiFi
          Issue Type: Bug
          Components: Core Framework
    Affects Versions: 1.1.0
            Reporter: Jeff Storck


In a cluster under load (4 nodes, ~80 active threads per node) requests for starting and stopping processors (and, for example, the timed UI thread requesting status) can fail due to socket timeouts during request replication.

{code}2017-01-10 17:46:21,878 INFO [StandardProcessScheduler Thread-8] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling GenerateFlowFile[id=87de49bc-0158-1000-6329-33d0dd20bd3a] to run
2017-01-10 17:46:21,921 WARN [Replicate Request Thread-9] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request PUT /nifi-api/processors/87dd1f69-0158-1000-0e85-aaf63af20c89 to m10wn03.local
domain:9091 due to {}
com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155) ~[jersey-client-1.19.jar:1.19]
        at com.sun.jersey.api.client.Client.handle(Client.java:652) ~[jersey-client-1.19.jar:1.19]
        at com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:123) ~[jersey-client-1.19.jar:1.19]
        at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682) ~[jersey-client-1.19.jar:1.19]
        at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) ~[jersey-client-1.19.jar:1.19]
        at com.sun.jersey.api.client.WebResource$Builder.put(WebResource.java:529) ~[jersey-client-1.19.jar:1.19]
        at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:590) ~[nifi-framework-cluster-1.1.0.jar:1.1.0]
        at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:770) ~[nifi-framework-cluster-1.1.0.jar:1.1.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_111]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_111]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.8.0_111]
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[na:1.8.0_111]
        at java.net.SocketInputStream.read(SocketInputStream.java:170) ~[na:1.8.0_111]
        at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[na:1.8.0_111]
        at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) ~[na:1.8.0_111]
        at sun.security.ssl.InputRecord.read(InputRecord.java:503) ~[na:1.8.0_111]
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) ~[na:1.8.0_111]
        at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930) ~[na:1.8.0_111]
        at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) ~[na:1.8.0_111]
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[na:1.8.0_111]
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[na:1.8.0_111]
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[na:1.8.0_111]
        at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704) ~[na:1.8.0_111]
        at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647) ~[na:1.8.0_111]
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569) ~[na:1.8.0_111]
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474) ~[na:1.8.0_111]
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) ~[na:1.8.0_111]
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338) ~[na:1.8.0_111]
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253) ~[jersey-client-1.19.jar:1.19]
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153) ~[jersey-client-1.19.jar:1.19]
        ... 12 common frames omitted{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)