You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Shing Hing Man <ma...@yahoo.com.INVALID> on 2015/06/01 23:11:03 UTC
Timeout in S3 Download using aws-s3 component
I am using Camel 2.15.2.
I am trying to download files from a S3 bucket at regular interval using the following aw-s3 route. String bucketName = "camel-lombok"; final String prefix = "out";
// Use RAW url encode the secretKey. String awsURLTemplate = "aws-s3://%s?accessKey=%s&secretKey=RAW(%s)®ion=eu-west-1&prefix=%s&deleteAfterRead=false&delay=5000&maxMessagesPerPoll=5"; final String awsURL = String.format(awsURLTemplate, bucketName, accessKey, secretKey, prefix);
DefaultCamelContext context = new DefaultCamelContext();
context.addRoutes(new RouteBuilder() { public void configure() { // Only download {bucket}/out/**.txt}. from(awsURL).filter().simple("${header.camelawss3key} regex '.*\\.txt'") .setHeader(Exchange.FILE_NAME, simple("${header.camelawss3key}")).to("file:/home/matmsh/tmp/camel/in");
} });
context.start();
Thread.sleep(600000);
But I get following timeout exception after about 1 minute.
216 [main] INFO org.apache.camel.impl.DefaultCamelContext - Apache Camel 2.15.2 (CamelContext: camel-1) is starting217 [main] INFO org.apache.camel.management.ManagedManagementStrategy - JMX is enabled474 [main] INFO org.apache.camel.impl.converter.DefaultTypeConverter - Loaded 233 type converters1703 [main] INFO org.apache.camel.impl.DefaultCamelContext - AllowUseOriginalMessage is enabled. If access to the original message is not needed, then its recommended to turn this option off as it may improve performance.1703 [main] INFO org.apache.camel.impl.DefaultCamelContext - StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at http://camel.apache.org/stream-caching.html1763 [main] INFO org.apache.camel.impl.DefaultCamelContext - Route: route1 started and consuming from: Endpoint[aws-s3://camel-lombok?accessKey=XXXX&delay=5000&deleteAfterRead=false&maxMessagesPerPoll=5&prefix=out®ion=eu-west-1&secretKey=xxxxxx]1763 [main] INFO org.apache.camel.impl.DefaultCamelContext - Total 1 routes, of which 1 is started.1764 [main] INFO org.apache.camel.impl.DefaultCamelContext - Apache Camel 2.15.2 (CamelContext: camel-1) started in 1.549 seconds[ead #0 - aws-s3://camel-lombok] AmazonHttpClient INFO Unable to execute HTTP request: Timeout waiting for connection from poolorg.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool at org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:226)[httpclient-4.3.4.jar:4.3.4] at org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:195)[httpclient-4.3.4.jar:4.3.4] at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_71] at java.lang.reflect.Method.invoke(Method.java:606)[:1.7.0_71] at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)[aws-java-sdk-1.8.9.1.jar:] at com.amazonaws.http.conn.$Proxy17.getConnection(Unknown Source)[:] at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:423)[httpclient-4.3.4.jar:4.3.4] at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)[httpclient-4.3.4.jar:4.3.4] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)[httpclient-4.3.4.jar:4.3.4] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)[httpclient-4.3.4.jar:4.3.4] at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:464)[aws-java-sdk-1.8.9.1.jar:] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:273)[aws-java-sdk-1.8.9.1.jar:] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3660)[aws-java-sdk-1.8.9.1.jar:] at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1133)[aws-java-sdk-1.8.9.1.jar:] at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1005)[aws-java-sdk-1.8.9.1.jar:] at org.apache.camel.component.aws.s3.S3Consumer.createExchanges(S3Consumer.java:114)[camel-aws-2.15.2.jar:2.15.2] at org.apache.camel.component.aws.s3.S3Consumer.poll(S3Consumer.java:95)[camel-aws-2.15.2.jar:2.15.2] at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:174)[camel-core-2.15.2.jar:2.15.2] at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:101)[camel-core-2.15.2.jar:2.15.2] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)[:1.7.0_71] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)[:1.7.0_71] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)[:1.7.0_71] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_71] at java.lang.Thread.run(Thread.java:745)[:1.7.0_71]
Thank advance for any assistance !Shing
Re: Timeout in S3 Download using aws-s3 component
Posted by Shing Hing Man <ma...@yahoo.com.INVALID>.
I have rerun the same piece of code in my last email. But this time, I get a different exception.
1952 [main] INFO org.apache.camel.impl.DefaultCamelContext - Apache Camel 2.15.2 (CamelContext: camel-1) started in 1.736 seconds2015-06-03 21:57:32,226 [://camel-lombok] INFO AmazonHttpClient - Unable to execute HTTP request: camel-lombok.s3.amazonaws.com failed to respondorg.apache.http.NoHttpResponseException: camel-lombok.s3.amazonaws.com failed to respond at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)[httpclient-4.3.4.jar:4.3.4] at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)[httpclient-4.3.4.jar:4.3.4] at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)[httpcore-4.3.2.jar:4.3.2] at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)[httpcore-4.3.2.jar:4.3.2] at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)[httpclient-4.3.4.jar:4.3.4] at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)[httpclient-4.3.4.jar:4.3.4] at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)[httpcore-4.3.2.jar:4.3.2] at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doReceiveResponse(SdkHttpRequestExecutor.java:66)[aws-java-sdk-1.8.9.1.jar:] at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)[httpcore-4.3.2.jar:4.3.2] at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)[httpclient-4.3.4.jar:4.3.4] at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)[httpclient-4.3.4.jar:4.3.4] at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)[httpclient-4.3.4.jar:4.3.4] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)[httpclient-4.3.4.jar:4.3.4] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)[httpclient-4.3.4.jar:4.3.4] at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:464)[aws-java-sdk-1.8.9.1.jar:] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:273)[aws-java-sdk-1.8.9.1.jar:] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3660)[aws-java-sdk-1.8.9.1.jar:] at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1133)[aws-java-sdk-1.8.9.1.jar:] at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1005)[aws-java-sdk-1.8.9.1.jar:] at org.apache.camel.component.aws.s3.S3Consumer.createExchanges(S3Consumer.java:114)[camel-aws-2.15.2.jar:2.15.2] at org.apache.camel.component.aws.s3.S3Consumer.poll(S3Consumer.java:95)[camel-aws-2.15.2.jar:2.15.2] at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:174)[camel-core-2.15.2.jar:2.15.2] at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:101)[camel-core-2.15.2.jar:2.15.2]
In the source code of S3Endpoint.java, in method
public Exchange createExchange(ExchangePattern pattern, S3Object s3Object)
message.setBody(s3Object.getObjectContent());
I can not find where s3Object.close() is called.
I wonder if my exception is related to s3Object.close() is not being called.
Shing
On Monday, 1 June 2015, 22:11, Shing Hing Man <ma...@yahoo.com.INVALID> wrote:
I am using Camel 2.15.2.
I am trying to download files from a S3 bucket at regular interval using the following aw-s3 route. String bucketName = "camel-lombok"; final String prefix = "out";
// Use RAW url encode the secretKey. String awsURLTemplate = "aws-s3://%s?accessKey=%s&secretKey=RAW(%s)®ion=eu-west-1&prefix=%s&deleteAfterRead=false&delay=5000&maxMessagesPerPoll=5"; final String awsURL = String.format(awsURLTemplate, bucketName, accessKey, secretKey, prefix);
DefaultCamelContext context = new DefaultCamelContext();
context.addRoutes(new RouteBuilder() { public void configure() { // Only download {bucket}/out/**.txt}. from(awsURL).filter().simple("${header.camelawss3key} regex '.*\\.txt'") .setHeader(Exchange.FILE_NAME, simple("${header.camelawss3key}")).to("file:/home/matmsh/tmp/camel/in");
} });
context.start();
Thread.sleep(600000);
But I get following timeout exception after about 1 minute.
216 [main] INFO org.apache.camel.impl.DefaultCamelContext - Apache Camel 2.15.2 (CamelContext: camel-1) is starting217 [main] INFO org.apache.camel.management.ManagedManagementStrategy - JMX is enabled474 [main] INFO org.apache.camel.impl.converter.DefaultTypeConverter - Loaded 233 type converters1703 [main] INFO org.apache.camel.impl.DefaultCamelContext - AllowUseOriginalMessage is enabled. If access to the original message is not needed, then its recommended to turn this option off as it may improve performance.1703 [main] INFO org.apache.camel.impl.DefaultCamelContext - StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at http://camel.apache.org/stream-caching.html1763 [main] INFO org.apache.camel.impl.DefaultCamelContext - Route: route1 started and consuming from: Endpoint[aws-s3://camel-lombok?accessKey=XXXX&delay=5000&deleteAfterRead=false&maxMessagesPerPoll=5&prefix=out®ion=eu-west-1&secretKey=xxxxxx]1763 [main] INFO org.apache.camel.impl.DefaultCamelContext - Total 1 routes, of which 1 is started.1764 [main] INFO org.apache.camel.impl.DefaultCamelContext - Apache Camel 2.15.2 (CamelContext: camel-1) started in 1.549 seconds[ead #0 - aws-s3://camel-lombok] AmazonHttpClient INFO Unable to execute HTTP request: Timeout waiting for connection from poolorg.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool at org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:226)[httpclient-4.3.4.jar:4.3.4] at org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:195)[httpclient-4.3.4.jar:4.3.4] at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_71] at java.lang.reflect.Method.invoke(Method.java:606)[:1.7.0_71] at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)[aws-java-sdk-1.8.9.1.jar:] at com.amazonaws.http.conn.$Proxy17.getConnection(Unknown Source)[:] at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:423)[httpclient-4.3.4.jar:4.3.4] at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)[httpclient-4.3.4.jar:4.3.4] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)[httpclient-4.3.4.jar:4.3.4] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)[httpclient-4.3.4.jar:4.3.4] at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:464)[aws-java-sdk-1.8.9.1.jar:] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:273)[aws-java-sdk-1.8.9.1.jar:] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3660)[aws-java-sdk-1.8.9.1.jar:] at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1133)[aws-java-sdk-1.8.9.1.jar:] at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1005)[aws-java-sdk-1.8.9.1.jar:] at org.apache.camel.component.aws.s3.S3Consumer.createExchanges(S3Consumer.java:114)[camel-aws-2.15.2.jar:2.15.2] at org.apache.camel.component.aws.s3.S3Consumer.poll(S3Consumer.java:95)[camel-aws-2.15.2.jar:2.15.2] at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:174)[camel-core-2.15.2.jar:2.15.2] at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:101)[camel-core-2.15.2.jar:2.15.2] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)[:1.7.0_71] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)[:1.7.0_71] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)[:1.7.0_71] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_71] at java.lang.Thread.run(Thread.java:745)[:1.7.0_71]
Thank advance for any assistance !Shing