You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Jeff Storck (JIRA)" <ji...@apache.org> on 2017/01/10 22:57:58 UTC
[jira] [Updated] (NIFI-3317) Under load,
ThreadPoolRequestReplicator is experiencing SocketTimeoutExceptions
[ https://issues.apache.org/jira/browse/NIFI-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Storck updated NIFI-3317:
------------------------------
Description:
In a cluster under load (4 nodes, ~80 active threads per node) requests for starting and stopping processors (and, for example, the timed UI thread requesting status) can fail due to socket timeouts during request replication. Sometimes, this results in a dialog shown on the UI that has a title of "Update Resource", but the dialog itself is empty.
{code}2017-01-10 17:46:21,878 INFO [StandardProcessScheduler Thread-8] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling GenerateFlowFile[id=87de49bc-0158-1000-6329-33d0dd20bd3a] to run
2017-01-10 17:46:21,921 WARN [Replicate Request Thread-9] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request PUT /nifi-api/processors/87dd1f69-0158-1000-0e85-aaf63af20c89 to m10wn03.local
domain:9091 due to {}
com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.Client.handle(Client.java:652) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:123) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.WebResource$Builder.put(WebResource.java:529) ~[jersey-client-1.19.jar:1.19]
at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:590) ~[nifi-framework-cluster-1.1.0.jar:1.1.0]
at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:770) ~[nifi-framework-cluster-1.1.0.jar:1.1.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.8.0_111]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[na:1.8.0_111]
at java.net.SocketInputStream.read(SocketInputStream.java:170) ~[na:1.8.0_111]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[na:1.8.0_111]
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) ~[na:1.8.0_111]
at sun.security.ssl.InputRecord.read(InputRecord.java:503) ~[na:1.8.0_111]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) ~[na:1.8.0_111]
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930) ~[na:1.8.0_111]
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) ~[na:1.8.0_111]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[na:1.8.0_111]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[na:1.8.0_111]
at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[na:1.8.0_111]
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704) ~[na:1.8.0_111]
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647) ~[na:1.8.0_111]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569) ~[na:1.8.0_111]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474) ~[na:1.8.0_111]
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) ~[na:1.8.0_111]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338) ~[na:1.8.0_111]
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153) ~[jersey-client-1.19.jar:1.19]
... 12 common frames omitted{code}
was:
In a cluster under load (4 nodes, ~80 active threads per node) requests for starting and stopping processors (and, for example, the timed UI thread requesting status) can fail due to socket timeouts during request replication.
{code}2017-01-10 17:46:21,878 INFO [StandardProcessScheduler Thread-8] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling GenerateFlowFile[id=87de49bc-0158-1000-6329-33d0dd20bd3a] to run
2017-01-10 17:46:21,921 WARN [Replicate Request Thread-9] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request PUT /nifi-api/processors/87dd1f69-0158-1000-0e85-aaf63af20c89 to m10wn03.local
domain:9091 due to {}
com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.Client.handle(Client.java:652) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:123) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.WebResource$Builder.put(WebResource.java:529) ~[jersey-client-1.19.jar:1.19]
at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:590) ~[nifi-framework-cluster-1.1.0.jar:1.1.0]
at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:770) ~[nifi-framework-cluster-1.1.0.jar:1.1.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.8.0_111]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[na:1.8.0_111]
at java.net.SocketInputStream.read(SocketInputStream.java:170) ~[na:1.8.0_111]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[na:1.8.0_111]
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) ~[na:1.8.0_111]
at sun.security.ssl.InputRecord.read(InputRecord.java:503) ~[na:1.8.0_111]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) ~[na:1.8.0_111]
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930) ~[na:1.8.0_111]
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) ~[na:1.8.0_111]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[na:1.8.0_111]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[na:1.8.0_111]
at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[na:1.8.0_111]
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704) ~[na:1.8.0_111]
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647) ~[na:1.8.0_111]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569) ~[na:1.8.0_111]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474) ~[na:1.8.0_111]
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) ~[na:1.8.0_111]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338) ~[na:1.8.0_111]
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153) ~[jersey-client-1.19.jar:1.19]
... 12 common frames omitted{code}
> Under load, ThreadPoolRequestReplicator is experiencing SocketTimeoutExceptions
> -------------------------------------------------------------------------------
>
> Key: NIFI-3317
> URL: https://issues.apache.org/jira/browse/NIFI-3317
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.1.0
> Reporter: Jeff Storck
>
> In a cluster under load (4 nodes, ~80 active threads per node) requests for starting and stopping processors (and, for example, the timed UI thread requesting status) can fail due to socket timeouts during request replication. Sometimes, this results in a dialog shown on the UI that has a title of "Update Resource", but the dialog itself is empty.
> {code}2017-01-10 17:46:21,878 INFO [StandardProcessScheduler Thread-8] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling GenerateFlowFile[id=87de49bc-0158-1000-6329-33d0dd20bd3a] to run
> 2017-01-10 17:46:21,921 WARN [Replicate Request Thread-9] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request PUT /nifi-api/processors/87dd1f69-0158-1000-0e85-aaf63af20c89 to m10wn03.local
> domain:9091 due to {}
> com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out
> at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155) ~[jersey-client-1.19.jar:1.19]
> at com.sun.jersey.api.client.Client.handle(Client.java:652) ~[jersey-client-1.19.jar:1.19]
> at com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:123) ~[jersey-client-1.19.jar:1.19]
> at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682) ~[jersey-client-1.19.jar:1.19]
> at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) ~[jersey-client-1.19.jar:1.19]
> at com.sun.jersey.api.client.WebResource$Builder.put(WebResource.java:529) ~[jersey-client-1.19.jar:1.19]
> at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:590) ~[nifi-framework-cluster-1.1.0.jar:1.1.0]
> at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:770) ~[nifi-framework-cluster-1.1.0.jar:1.1.0]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_111]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_111]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
> Caused by: java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.8.0_111]
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[na:1.8.0_111]
> at java.net.SocketInputStream.read(SocketInputStream.java:170) ~[na:1.8.0_111]
> at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[na:1.8.0_111]
> at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) ~[na:1.8.0_111]
> at sun.security.ssl.InputRecord.read(InputRecord.java:503) ~[na:1.8.0_111]
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) ~[na:1.8.0_111]
> at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930) ~[na:1.8.0_111]
> at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) ~[na:1.8.0_111]
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[na:1.8.0_111]
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[na:1.8.0_111]
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[na:1.8.0_111]
> at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704) ~[na:1.8.0_111]
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647) ~[na:1.8.0_111]
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569) ~[na:1.8.0_111]
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474) ~[na:1.8.0_111]
> at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) ~[na:1.8.0_111]
> at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338) ~[na:1.8.0_111]
> at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253) ~[jersey-client-1.19.jar:1.19]
> at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153) ~[jersey-client-1.19.jar:1.19]
> ... 12 common frames omitted{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)