You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Shing Hing Man <ma...@yahoo.com.INVALID> on 2015/01/10 15:43:09 UTC
Timeout Exception in S3 download
I am using Apache Camel 2.14.1. I have set up the following S3 route to download from a S3 Bucket.
String bucketName = "myBucket"; String prefix = "out"; String awsURLTemplate="aws-s3://%s?accessKey=%s&secretKey=RAW(%s)®ion=eu-west-1&prefix=%s&deleteAfterRead=false"; final String awsURL = String.format(awsURLTemplate,bucketName, accessKey,secretKey,prefix); DefaultCamelContext context = new DefaultCamelContext(); context.addRoutes(new RouteBuilder() { public void configure() { // Only download {myBucket}/out/**.txt}. from(awsURL).filter().simple("${header.camelawss3key} regex '.*\\.txt'") .setHeader(Exchange.FILE_NAME, simple("${header.camelawss3key}")).to("file:camel/in"); } }); context.start(); Thread.sleep(600000);
When the above is run, all the txt files in S3 myBucket/out are downloaded. But about 1 minute later (without adding any files to the bucket), I get the following timeout exception.[ead #0 - aws-s3://camel-lombok] AmazonHttpClient INFO Unable to execute HTTP request: Timeout waiting for connection from poolorg.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool at org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:226)[httpclient-4.3.4.jar:4.3.4] at org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:195)[httpclient-4.3.4.jar:4.3.4] at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_40] at java.lang.reflect.Method.invoke(Method.java:606)[:1.7.0_40] at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)[aws-java-sdk-1.8.9.1.jar:] at com.amazonaws.http.conn.$Proxy13.getConnection(Unknown Source)[:] at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:423)[httpclient-4.3.4.jar:4.3.4] at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)[httpclient-4.3.4.jar:4.3.4] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)[httpclient-4.3.4.jar:4.3.4] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)[httpclient-4.3.4.jar:4.3.4] at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:464)[aws-java-sdk-1.8.9.1.jar:] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:273)[aws-java-sdk-1.8.9.1.jar:] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3660)[aws-java-sdk-1.8.9.1.jar:] at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1133)[aws-java-sdk-1.8.9.1.jar:] at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1005)[aws-java-sdk-1.8.9.1.jar:] at org.apache.camel.component.aws.s3.S3Consumer.createExchanges(S3Consumer.java:104)[camel-aws-2.14.1.jar:2.14.1] at org.apache.camel.component.aws.s3.S3Consumer.poll(S3Consumer.java:85)[camel-aws-2.14.1.jar:2.14.1] at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187)[camel-core-2.14.1.jar:2.14.1] at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:114)[camel-core-2.14.1.jar:2.14.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)[:1.7.0_40] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)[:1.7.0_40] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)[:1.7.0_40] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.7.0_40] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_40] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_40] at java.lang.Thread.run(Thread.java:724)[:1.7.0_40]
Thanks in advance for any assistance ! Shing
Re: Timeout Exception in S3 download
Posted by Shing Hing Man <ma...@yahoo.com.INVALID>.
I've added the delay and maxMessagePerPoll, as you suggested, to the path, but I still get the same timeout exception.
Shing
On Monday, 12 January 2015, 2:10, Willem Jiang <wi...@gmail.com> wrote:
Can you increase the delay option and maxMessagePerPoll to see if the error is still there?
from("aws-s3://MyBucket?amazonS3Client=#client&delay=5000&maxMessagesPerPoll=5")
.to("mock:result”);
--
Willem Jiang
Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On January 10, 2015 at 10:44:44 PM, Shing Hing Man (matmsh@yahoo.com.invalid) wrote:
>
> I am using Apache Camel 2.14.1. I have set up the following S3 route to download from a S3
> Bucket.
> String bucketName = "myBucket"; String prefix = "out"; String awsURLTemplate="aws-s3://%s?accessKey=%s&secretKey=RAW(%s)®ion=eu-west-1&prefix=%s&deleteAfterRead=false";
> final String awsURL = String.format(awsURLTemplate,bucketName, accessKey,secretKey,prefix);
> DefaultCamelContext context = new DefaultCamelContext(); context.addRoutes(new
> RouteBuilder() { public void configure() { // Only download {myBucket}/out/**.txt}.
> from(awsURL).filter().simple("${header.camelawss3key} regex '.*\\.txt'") .setHeader(Exchange.FILE_NAME,
> simple("${header.camelawss3key}")).to("file:camel/in"); } }); context.start();
> Thread.sleep(600000);
> When the above is run, all the txt files in S3 myBucket/out are downloaded. But about 1
> minute later (without adding any files to the bucket), I get the following timeout exception.[ead
> #0 - aws-s3://camel-lombok] AmazonHttpClient INFO Unable to execute HTTP request:
> Timeout waiting for connection from poolorg.apache.http.conn.ConnectionPoolTimeoutException:
> Timeout waiting for connection from pool at org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:226)[httpclient-4.3.4.jar:4.3.4]
> at org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:195)[httpclient-4.3.4.jar:4.3.4]
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_40]
> at java.lang.reflect.Method.invoke(Method.java:606)[:1.7.0_40] at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)[aws-java-sdk-1.8.9.1.jar:]
> at com.amazonaws.http.conn.$Proxy13.getConnection(Unknown Source)[:] at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:423)[httpclient-4.3.4.jar:4.3.4]
> at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)[httpclient-4.3.4.jar:4.3.4]
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)[httpclient-4.3.4.jar:4.3.4]
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)[httpclient-4.3.4.jar:4.3.4]
> at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:464)[aws-java-sdk-1.8.9.1.jar:]
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:273)[aws-java-sdk-1.8.9.1.jar:]
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3660)[aws-java-sdk-1.8.9.1.jar:]
> at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1133)[aws-java-sdk-1.8.9.1.jar:]
> at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1005)[aws-java-sdk-1.8.9.1.jar:]
> at org.apache.camel.component.aws.s3.S3Consumer.createExchanges(S3Consumer.java:104)[camel-aws-2.14.1.jar:2.14.1]
> at org.apache.camel.component.aws.s3.S3Consumer.poll(S3Consumer.java:85)[camel-aws-2.14.1.jar:2.14.1]
> at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187)[camel-core-2.14.1.jar:2.14.1]
> at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:114)[camel-core-2.14.1.jar:2.14.1]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)[:1.7.0_40]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)[:1.7.0_40]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)[:1.7.0_40]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.7.0_40]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_40]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_40]
> at java.lang.Thread.run(Thread.java:724)[:1.7.0_40]
> Thanks in advance for any assistance ! Shing
>
>
Re: Timeout Exception in S3 download
Posted by Willem Jiang <wi...@gmail.com>.
Can you increase the delay option and maxMessagePerPoll to see if the error is still there?
from("aws-s3://MyBucket?amazonS3Client=#client&delay=5000&maxMessagesPerPoll=5")
.to("mock:result”);
--
Willem Jiang
Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On January 10, 2015 at 10:44:44 PM, Shing Hing Man (matmsh@yahoo.com.invalid) wrote:
>
> I am using Apache Camel 2.14.1. I have set up the following S3 route to download from a S3
> Bucket.
> String bucketName = "myBucket"; String prefix = "out"; String awsURLTemplate="aws-s3://%s?accessKey=%s&secretKey=RAW(%s)®ion=eu-west-1&prefix=%s&deleteAfterRead=false";
> final String awsURL = String.format(awsURLTemplate,bucketName, accessKey,secretKey,prefix);
> DefaultCamelContext context = new DefaultCamelContext(); context.addRoutes(new
> RouteBuilder() { public void configure() { // Only download {myBucket}/out/**.txt}.
> from(awsURL).filter().simple("${header.camelawss3key} regex '.*\\.txt'") .setHeader(Exchange.FILE_NAME,
> simple("${header.camelawss3key}")).to("file:camel/in"); } }); context.start();
> Thread.sleep(600000);
> When the above is run, all the txt files in S3 myBucket/out are downloaded. But about 1
> minute later (without adding any files to the bucket), I get the following timeout exception.[ead
> #0 - aws-s3://camel-lombok] AmazonHttpClient INFO Unable to execute HTTP request:
> Timeout waiting for connection from poolorg.apache.http.conn.ConnectionPoolTimeoutException:
> Timeout waiting for connection from pool at org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:226)[httpclient-4.3.4.jar:4.3.4]
> at org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:195)[httpclient-4.3.4.jar:4.3.4]
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_40]
> at java.lang.reflect.Method.invoke(Method.java:606)[:1.7.0_40] at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)[aws-java-sdk-1.8.9.1.jar:]
> at com.amazonaws.http.conn.$Proxy13.getConnection(Unknown Source)[:] at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:423)[httpclient-4.3.4.jar:4.3.4]
> at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)[httpclient-4.3.4.jar:4.3.4]
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)[httpclient-4.3.4.jar:4.3.4]
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)[httpclient-4.3.4.jar:4.3.4]
> at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:464)[aws-java-sdk-1.8.9.1.jar:]
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:273)[aws-java-sdk-1.8.9.1.jar:]
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3660)[aws-java-sdk-1.8.9.1.jar:]
> at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1133)[aws-java-sdk-1.8.9.1.jar:]
> at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1005)[aws-java-sdk-1.8.9.1.jar:]
> at org.apache.camel.component.aws.s3.S3Consumer.createExchanges(S3Consumer.java:104)[camel-aws-2.14.1.jar:2.14.1]
> at org.apache.camel.component.aws.s3.S3Consumer.poll(S3Consumer.java:85)[camel-aws-2.14.1.jar:2.14.1]
> at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187)[camel-core-2.14.1.jar:2.14.1]
> at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:114)[camel-core-2.14.1.jar:2.14.1]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)[:1.7.0_40]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)[:1.7.0_40]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)[:1.7.0_40]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.7.0_40]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_40]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_40]
> at java.lang.Thread.run(Thread.java:724)[:1.7.0_40]
> Thanks in advance for any assistance ! Shing
>
>