FetchS3Object throwing com.amazonaws.SdkClientException

[Wyll Ingersoll <wy...@keepertech.com>]

Using Nifi 1.11.4, the FetchS3Object processor occasionally throws the following error:

routing to failure: com.amazonaws.SdkClientException: Unable to execute HTTP request: The target server failed to respond.
The full stack trace looks like:


com.amazonaws.SdkClientException: Unable to execute HTTP request: The target server failed to respond

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1175)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1121)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)

at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)

at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)

at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)

at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4926)

at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4872)

at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1472)

at org.apache.nifi.processors.aws.s3.FetchS3Object.onTrigger(FetchS3Object.java:159)

at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)

at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)

at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)

at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)

at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: org.apache.http.NoHttpResponseException: The target server failed to respond

at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)

at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)

at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)

at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)

at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)

at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)

at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doReceiveResponse(SdkHttpRequestExecutor.java:82)

at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)

at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)

at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)

at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)

at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)

at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)

at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1297)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)

... 23 common frames omitted



The timeout period is plenty long enough and the server is definitely available.  Not sure what would be causing the problem but there were some bugs in the Apache HTTPClient module that are referenced in some forums discussing a similar issue - https://issues.apache.org/jira/browse/HTTPCLIENT-1610

Is this a known problem in the Nifi processor, perhaps using a buggy HTTP Client library? It only happens occasionally, usually when the processor is under heavy load. Looking for suggestions/workarounds.

Thanks
   Wyllys Ingersoll