You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by kretin <kr...@mac.com> on 2018/01/05 02:19:25 UTC
Problem with camel-aws S3 when multiPartUpload is set to true
I created a simple camel route to poll for files in a local directory and upload them to a Ceph (S3) server at my University. I am using apache camel 2.20.0 with the camel-aws S3 component, when I set multiPartUpload=false (the default) in the uri, everything works fine, but if I change to multiPartUpload=true, it fails.
I know there is nothing wrong with my s3 secret or s3 access key because when I set multiPartUpload=false, everything works (there are no crazy plus (+) characters that need to be escaped in the keys).
Here is the stack trace:
com.amazonaws.services.s3.model.AmazonS3Exception: null (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: tx000000000000002e9edee-005a4ed3d2-2213a2-uky-campus-1; S3 Extended Request ID: 2213a2-uky-campus-1-uky)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1592) ~[aws-java-sdk-core-1.11.186.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1257) ~[aws-java-sdk-core-1.11.186.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1029) ~[aws-java-sdk-core-1.11.186.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741) ~[aws-java-sdk-core-1.11.186.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715) ~[aws-java-sdk-core-1.11.186.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697) ~[aws-java-sdk-core-1.11.186.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665) ~[aws-java-sdk-core-1.11.186.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647) ~[aws-java-sdk-core-1.11.186.jar:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511) ~[aws-java-sdk-core-1.11.186.jar:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4227) ~[aws-java-sdk-s3-1.11.186.jar:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4174) ~[aws-java-sdk-s3-1.11.186.jar:?]
at com.amazonaws.services.s3.AmazonS3Client.abortMultipartUpload(AmazonS3Client.java:2928) ~[aws-java-sdk-s3-1.11.186.jar:?]
at org.apache.camel.component.aws.s3.S3Producer.processMultiPart(S3Producer.java:181) ~[camel-aws-2.20.0.jar:2.20.0]
at org.apache.camel.component.aws.s3.S3Producer.process(S3Producer.java:84) ~[camel-aws-2.20.0.jar:2.20.0]
at ...
My camel-context.xml looks like:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd">
<bean id="properties"
class="org.apache.camel.component.properties.PropertiesComponent">
<property name="location" value="config.properties" />
</bean>
<bean id="bridgePropertyPlaceholder"
class="org.apache.camel.spring.spi.BridgePropertyPlaceholderConfigurer">
<property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_OVERRIDE" />
<property name="location" value="config.properties" />
</bean>
<bean id="amazonClient" class="com.amazonaws.services.s3.AmazonS3Client">
<constructor-arg>
<bean class="com.amazonaws.auth.BasicAWSCredentials">
<constructor-arg name="accessKey" value="${s3AccessKey}" />
<constructor-arg name="secretKey" value="${s3SecretKey}" />
</bean>
</constructor-arg>
<property name="s3ClientOptions">
<bean class="com.amazonaws.services.s3.S3ClientOptions">
<property name="pathStyleAccess" value="true" />
</bean>
</property>
<property name="endpoint" value="${s3Endpoint}" />
</bean>
<camelContext xmlns="http://camel.apache.org/schema/spring">
<route>
<from uri="file:target/sendToS3/?antInclude=*.*" />
<log message="Found file: ${in.header.CamelFileName}" />
<setHeader headerName="CamelAwsS3Key">
<simple>testMultiPart/${in.header.CamelFileName}
</simple>
</setHeader>
<setHeader headerName="CamelAwsS3ContentLength">
<simple>${in.header.CamelFileLength}
</simple>
</setHeader>
<log message="Send file to S3: ${properties:s3Endpoint}" />
<to uri="aws-s3://{{s3Bucket}}?amazonS3Client=#amazonClient&multiPartUpload=true" />
<log message="Done sending file." />
</route>
</camelContext>
</beans>
Again, everything works fine if I set multiPartUpload=false in the above camel-context.xml
I have tried a lot of things like:
• setting the CamelAwsS3ContentMD5 header to the MD5 hash of the file (which doesn't make sense for multi-part files)
• various settings for the partSize parameter
• different sized files from very large to very small
• setting the system parameter: System.setProperty("com.amazonaws.services.s3.disablePutObjectMD5Validation", "true");
If I turn on trace debugging for camel, it doesnt help much:
[d #2 - file://target/sendToS3/] S3Producer TRACE Initiating multipart upload [com.amazonaws.services.s3.model.InitiateMultipartUploadRequest@3731147a] from exchange [Exchange[ID-Toucan-local-1515115111374-0-1]]...
[d #2 - file://target/sendToS3/] S3Producer TRACE Uploading part [1] for testMultiPart/testfile.zip
[d #2 - file://target/sendToS3/] DefaultErrorHandler TRACE Is exchangeId: ID-Toucan-local-1515115111374-0-1 interrupted? false
[d #2 - file://target/sendToS3/] DefaultErrorHandler TRACE Is exchangeId: ID-Toucan-local-1515115111374-0-1 done? false
[d #2 - file://target/sendToS3/] DefaultErrorHandler TRACE isRunAllowed() -> true (Run allowed if we are not stopped/stopping)
[d #2 - file://target/sendToS3/] DefaultExceptionPolicyStrategy TRACE Finding best suited exception policy for thrown exception com.amazonaws.services.s3.model.AmazonS3Exception
[d #2 - file://target/sendToS3/] DefaultExceptionPolicyStrategy TRACE Finding best suited exception policy for thrown exception com.amazonaws.services.s3.model.AmazonS3Exception
[d #2 - file://target/sendToS3/] DefaultExceptionPolicyStrategy TRACE Found 0 candidates
[d #2 - file://target/sendToS3/] DefaultErrorHandler DEBUG Failed delivery for (MessageId: ID-Toucan-local-1515115111374-0-2 on ExchangeId: ID-Toucan-local-1515115111374-0-1). On delivery attempt: 0 caught: com.amazonaws.services.s3.model.AmazonS3Exception: null (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: tx000000000000002e9edee-005a4ed3d2-2213a2-uky-campus-1; S3 Extended Request ID: 2213a2-uky-campus-1-uky), S3 Extended Request ID: 2213a2-uky-campus-1-uky
[d #2 - file://target/sendToS3/] DefaultErrorHandler TRACE isRedeliveryAllowed() -> true (we are not stopping/stopped)
[d #2 - file://target/sendToS3/] DefaultErrorHandler TRACE This exchange is not handled or continued so its marked as failed: Exchange[ID-Toucan-local-1515115111374-0-1]
[d #2 - file://target/sendToS3/] InstrumentationProcessor TRACE to: Recording duration: 356930 millis for exchange: Exchange[ID-Toucan-local-1515115111374-0-1]
[d #2 - file://target/sendToS3/] DefaultErrorHandler ERROR Failed delivery for (MessageId: ID-Toucan-local-1515115111374-0-2 on ExchangeId: ID-Toucan-local-1515115111374-0-1). Exhausted after delivery attempt: 1 caught: com.amazonaws.services.s3.model.AmazonS3Exception: null (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: tx000000000000002e9edee-005a4ed3d2-2213a2-uky-campus-1; S3 Extended Request ID: 2213a2-uky-campus-1-uky), S3 Extended Request ID: 2213a2-uky-campus-1-uky
The reason I need multi-part uploads to work is for very large files, but I can't get it to work on any file size.
(I posted this on StackOverflow since I am not sure how active this mailing list is: https://stackoverflow.com/questions/48106180/apache-camel-s3-upload-fails-with-status-code-403-signaturedoesnotmatch-when-mu <https://stackoverflow.com/questions/48106180/apache-camel-s3-upload-fails-with-status-code-403-signaturedoesnotmatch-when-mu>)
Re: Problem with camel-aws S3 when multiPartUpload is set to true
Posted by Andrea Cosentino <an...@yahoo.com.INVALID>.
If you are passing the client as bean from registry, it makes no sense to define the region, you can set the region directly on the client you're passing.
--
Andrea Cosentino
----------------------------------
Apache Camel PMC Member
Apache Karaf Committer
Apache Servicemix PMC Member
Email: ancosen1985@yahoo.com
Twitter: @oscerd2
Github: oscerd
On Friday, January 5, 2018, 2:24:53 PM GMT+1, kretin <kr...@mac.com> wrote:
I tried adding a region like this and it still fails with the same error:
<to uri="aws-s3://{{s3Bucket}}?amazonS3Client=#amazonClient&multiPartUpload=true&region=us-east-1" />
> On Jan 5, 2018, at 4:23 AM, Artur Jablonski <aj...@ravenpack.com> wrote:
>
> Try setting region on your AmazonS3Client bean.
>
> On Fri, Jan 5, 2018 at 3:19 AM, kretin <kr...@mac.com> wrote:
>
>> I created a simple camel route to poll for files in a local directory and
>> upload them to a Ceph (S3) server at my University. I am using apache camel
>> 2.20.0 with the camel-aws S3 component, when I set multiPartUpload=false
>> (the default) in the uri, everything works fine, but if I change to
>> multiPartUpload=true, it fails.
>>
>> I know there is nothing wrong with my s3 secret or s3 access key because
>> when I set multiPartUpload=false, everything works (there are no crazy plus
>> (+) characters that need to be escaped in the keys).
>>
>> Here is the stack trace:
>>
>> com.amazonaws.services.s3.model.AmazonS3Exception: null (Service: Amazon
>> S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID:
>> tx000000000000002e9edee-005a4ed3d2-2213a2-uky-campus-1; S3 Extended
>> Request ID: 2213a2-uky-campus-1-uky)
>> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.
>> handleErrorResponse(AmazonHttpClient.java:1592)
>> ~[aws-java-sdk-core-1.11.186.jar:?]
>> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.
>> executeOneRequest(AmazonHttpClient.java:1257)
>> ~[aws-java-sdk-core-1.11.186.jar:?]
>> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1029)
>> ~[aws-java-sdk-core-1.11.186.jar:?]
>> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741)
>> ~[aws-java-sdk-core-1.11.186.jar:?]
>> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.
>> executeWithTimer(AmazonHttpClient.java:715) ~[aws-java-sdk-core-1.11.186.
>> jar:?]
>> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697)
>> ~[aws-java-sdk-core-1.11.186.jar:?]
>> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665)
>> ~[aws-java-sdk-core-1.11.186.jar:?]
>> at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.
>> execute(AmazonHttpClient.java:647) ~[aws-java-sdk-core-1.11.186.jar:?]
>> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511)
>> ~[aws-java-sdk-core-1.11.186.jar:?]
>> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4227)
>> ~[aws-java-sdk-s3-1.11.186.jar:?]
>> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4174)
>> ~[aws-java-sdk-s3-1.11.186.jar:?]
>> at com.amazonaws.services.s3.AmazonS3Client.abortMultipartUpload(AmazonS3Client.java:2928)
>> ~[aws-java-sdk-s3-1.11.186.jar:?]
>> at org.apache.camel.component.aws.s3.S3Producer.
>> processMultiPart(S3Producer.java:181) ~[camel-aws-2.20.0.jar:2.20.0]
>> at org.apache.camel.component.aws.s3.S3Producer.process(S3Producer.java:84)
>> ~[camel-aws-2.20.0.jar:2.20.0]
>> at ...
>>
>> My camel-context.xml looks like:
>>
>> <?xml version="1.0" encoding="UTF-8"?>
>> <beans xmlns="http://www.springframework.org/schema/beans"
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> xsi:schemaLocation="
>> http://www.springframework.org/schema/beans
>> http://www.springframework.org/schema/beans/spring-beans.xsd
>> http://camel.apache.org/schema/spring http://camel.apache.org/
>> schema/spring/camel-spring.xsd">
>>
>> <bean id="properties"
>> class="org.apache.camel.component.properties.PropertiesComponent">
>> <property name="location" value="config.properties" />
>> </bean>
>>
>> <bean id="bridgePropertyPlaceholder"
>> class="org.apache.camel.spring.spi.BridgePropertyPlaceholderConfi
>> gurer">
>> <property name="systemPropertiesModeName"
>> value="SYSTEM_PROPERTIES_MODE_OVERRIDE" />
>> <property name="location" value="config.properties" />
>> </bean>
>>
>> <bean id="amazonClient" class="com.amazonaws.services.
>> s3.AmazonS3Client">
>> <constructor-arg>
>> <bean class="com.amazonaws.auth.BasicAWSCredentials">
>> <constructor-arg name="accessKey" value="${s3AccessKey}" />
>> <constructor-arg name="secretKey" value="${s3SecretKey}" />
>> </bean>
>> </constructor-arg>
>> <property name="s3ClientOptions">
>> <bean class="com.amazonaws.services.s3.S3ClientOptions">
>> <property name="pathStyleAccess" value="true" />
>> </bean>
>> </property>
>> <property name="endpoint" value="${s3Endpoint}" />
>> </bean>
>>
>> <camelContext xmlns="http://camel.apache.org/schema/spring">
>> <route>
>> <from uri="file:target/sendToS3/?antInclude=*.*" />
>>
>> <log message="Found file: ${in.header.CamelFileName}" />
>>
>> <setHeader headerName="CamelAwsS3Key">
>> <simple>testMultiPart/${in.header.CamelFileName}
>> </simple>
>> </setHeader>
>>
>> <setHeader headerName="CamelAwsS3ContentLength">
>> <simple>${in.header.CamelFileLength}
>> </simple>
>> </setHeader>
>>
>> <log message="Send file to S3: ${properties:s3Endpoint}" />
>>
>> <to uri="aws-s3://{{s3Bucket}}?amazonS3Client=#amazonClient&multiPartUpload=true"
>> />
>>
>> <log message="Done sending file." />
>>
>> </route>
>> </camelContext>
>> </beans>
>>
>> Again, everything works fine if I set multiPartUpload=false in the above
>> camel-context.xml
>>
>> I have tried a lot of things like:
>>
>> • setting the CamelAwsS3ContentMD5 header to the MD5 hash of the
>> file (which doesn't make sense for multi-part files)
>> • various settings for the partSize parameter
>> • different sized files from very large to very small
>> • setting the system parameter: System.setProperty("com.
>> amazonaws.services.s3.disablePutObjectMD5Validation", "true");
>>
>> If I turn on trace debugging for camel, it doesnt help much:
>>
>> [d #2 - file://target/sendToS3/] S3Producer TRACE
>> Initiating multipart upload [com.amazonaws.services.s3.model.
>> InitiateMultipartUploadRequest@3731147a] from exchange
>> [Exchange[ID-Toucan-local-1515115111374-0-1]]...
>> [d #2 - file://target/sendToS3/] S3Producer TRACE
>> Uploading part [1] for testMultiPart/testfile.zip
>> [d #2 - file://target/sendToS3/] DefaultErrorHandler TRACE Is
>> exchangeId: ID-Toucan-local-1515115111374-0-1 interrupted? false
>> [d #2 - file://target/sendToS3/] DefaultErrorHandler TRACE Is
>> exchangeId: ID-Toucan-local-1515115111374-0-1 done? false
>> [d #2 - file://target/sendToS3/] DefaultErrorHandler TRACE
>> isRunAllowed() -> true (Run allowed if we are not stopped/stopping)
>> [d #2 - file://target/sendToS3/] DefaultExceptionPolicyStrategy TRACE
>> Finding best suited exception policy for thrown exception
>> com.amazonaws.services.s3.model.AmazonS3Exception
>> [d #2 - file://target/sendToS3/] DefaultExceptionPolicyStrategy TRACE
>> Finding best suited exception policy for thrown exception
>> com.amazonaws.services.s3.model.AmazonS3Exception
>> [d #2 - file://target/sendToS3/] DefaultExceptionPolicyStrategy TRACE
>> Found 0 candidates
>> [d #2 - file://target/sendToS3/] DefaultErrorHandler DEBUG
>> Failed delivery for (MessageId: ID-Toucan-local-1515115111374-0-2 on
>> ExchangeId: ID-Toucan-local-1515115111374-0-1). On delivery attempt: 0
>> caught: com.amazonaws.services.s3.model.AmazonS3Exception: null (Service:
>> Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID:
>> tx000000000000002e9edee-005a4ed3d2-2213a2-uky-campus-1; S3 Extended
>> Request ID: 2213a2-uky-campus-1-uky), S3 Extended Request ID:
>> 2213a2-uky-campus-1-uky
>> [d #2 - file://target/sendToS3/] DefaultErrorHandler TRACE
>> isRedeliveryAllowed() -> true (we are not stopping/stopped)
>> [d #2 - file://target/sendToS3/] DefaultErrorHandler TRACE This
>> exchange is not handled or continued so its marked as failed:
>> Exchange[ID-Toucan-local-1515115111374-0-1]
>> [d #2 - file://target/sendToS3/] InstrumentationProcessor TRACE to:
>> Recording duration: 356930 millis for exchange: Exchange[ID-Toucan-local-
>> 1515115111374-0-1]
>> [d #2 - file://target/sendToS3/] DefaultErrorHandler ERROR
>> Failed delivery for (MessageId: ID-Toucan-local-1515115111374-0-2 on
>> ExchangeId: ID-Toucan-local-1515115111374-0-1). Exhausted after delivery
>> attempt: 1 caught: com.amazonaws.services.s3.model.AmazonS3Exception:
>> null (Service: Amazon S3; Status Code: 403; Error Code:
>> SignatureDoesNotMatch; Request ID: tx000000000000002e9edee-
>> 005a4ed3d2-2213a2-uky-campus-1; S3 Extended Request ID:
>> 2213a2-uky-campus-1-uky), S3 Extended Request ID: 2213a2-uky-campus-1-uky
>>
>> The reason I need multi-part uploads to work is for very large files, but
>> I can't get it to work on any file size.
>>
>> (I posted this on StackOverflow since I am not sure how active this
>> mailing list is: https://stackoverflow.com/questions/48106180/apache-
>> camel-s3-upload-fails-with-status-code-403-signaturedoesnotmatch-when-mu <
>> https://stackoverflow.com/questions/48106180/apache-
>> camel-s3-upload-fails-with-status-code-403-signaturedoesnotmatch-when-mu>)
>>
>>
>>
Re: Problem with camel-aws S3 when multiPartUpload is set to true
Posted by kretin <kr...@mac.com>.
I tried adding a region like this and it still fails with the same error:
<to uri="aws-s3://{{s3Bucket}}?amazonS3Client=#amazonClient&multiPartUpload=true&region=us-east-1" />
> On Jan 5, 2018, at 4:23 AM, Artur Jablonski <aj...@ravenpack.com> wrote:
>
> Try setting region on your AmazonS3Client bean.
>
> On Fri, Jan 5, 2018 at 3:19 AM, kretin <kr...@mac.com> wrote:
>
>> I created a simple camel route to poll for files in a local directory and
>> upload them to a Ceph (S3) server at my University. I am using apache camel
>> 2.20.0 with the camel-aws S3 component, when I set multiPartUpload=false
>> (the default) in the uri, everything works fine, but if I change to
>> multiPartUpload=true, it fails.
>>
>> I know there is nothing wrong with my s3 secret or s3 access key because
>> when I set multiPartUpload=false, everything works (there are no crazy plus
>> (+) characters that need to be escaped in the keys).
>>
>> Here is the stack trace:
>>
>> com.amazonaws.services.s3.model.AmazonS3Exception: null (Service: Amazon
>> S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID:
>> tx000000000000002e9edee-005a4ed3d2-2213a2-uky-campus-1; S3 Extended
>> Request ID: 2213a2-uky-campus-1-uky)
>> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.
>> handleErrorResponse(AmazonHttpClient.java:1592)
>> ~[aws-java-sdk-core-1.11.186.jar:?]
>> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.
>> executeOneRequest(AmazonHttpClient.java:1257)
>> ~[aws-java-sdk-core-1.11.186.jar:?]
>> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1029)
>> ~[aws-java-sdk-core-1.11.186.jar:?]
>> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741)
>> ~[aws-java-sdk-core-1.11.186.jar:?]
>> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.
>> executeWithTimer(AmazonHttpClient.java:715) ~[aws-java-sdk-core-1.11.186.
>> jar:?]
>> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697)
>> ~[aws-java-sdk-core-1.11.186.jar:?]
>> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665)
>> ~[aws-java-sdk-core-1.11.186.jar:?]
>> at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.
>> execute(AmazonHttpClient.java:647) ~[aws-java-sdk-core-1.11.186.jar:?]
>> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511)
>> ~[aws-java-sdk-core-1.11.186.jar:?]
>> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4227)
>> ~[aws-java-sdk-s3-1.11.186.jar:?]
>> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4174)
>> ~[aws-java-sdk-s3-1.11.186.jar:?]
>> at com.amazonaws.services.s3.AmazonS3Client.abortMultipartUpload(AmazonS3Client.java:2928)
>> ~[aws-java-sdk-s3-1.11.186.jar:?]
>> at org.apache.camel.component.aws.s3.S3Producer.
>> processMultiPart(S3Producer.java:181) ~[camel-aws-2.20.0.jar:2.20.0]
>> at org.apache.camel.component.aws.s3.S3Producer.process(S3Producer.java:84)
>> ~[camel-aws-2.20.0.jar:2.20.0]
>> at ...
>>
>> My camel-context.xml looks like:
>>
>> <?xml version="1.0" encoding="UTF-8"?>
>> <beans xmlns="http://www.springframework.org/schema/beans"
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> xsi:schemaLocation="
>> http://www.springframework.org/schema/beans
>> http://www.springframework.org/schema/beans/spring-beans.xsd
>> http://camel.apache.org/schema/spring http://camel.apache.org/
>> schema/spring/camel-spring.xsd">
>>
>> <bean id="properties"
>> class="org.apache.camel.component.properties.PropertiesComponent">
>> <property name="location" value="config.properties" />
>> </bean>
>>
>> <bean id="bridgePropertyPlaceholder"
>> class="org.apache.camel.spring.spi.BridgePropertyPlaceholderConfi
>> gurer">
>> <property name="systemPropertiesModeName"
>> value="SYSTEM_PROPERTIES_MODE_OVERRIDE" />
>> <property name="location" value="config.properties" />
>> </bean>
>>
>> <bean id="amazonClient" class="com.amazonaws.services.
>> s3.AmazonS3Client">
>> <constructor-arg>
>> <bean class="com.amazonaws.auth.BasicAWSCredentials">
>> <constructor-arg name="accessKey" value="${s3AccessKey}" />
>> <constructor-arg name="secretKey" value="${s3SecretKey}" />
>> </bean>
>> </constructor-arg>
>> <property name="s3ClientOptions">
>> <bean class="com.amazonaws.services.s3.S3ClientOptions">
>> <property name="pathStyleAccess" value="true" />
>> </bean>
>> </property>
>> <property name="endpoint" value="${s3Endpoint}" />
>> </bean>
>>
>> <camelContext xmlns="http://camel.apache.org/schema/spring">
>> <route>
>> <from uri="file:target/sendToS3/?antInclude=*.*" />
>>
>> <log message="Found file: ${in.header.CamelFileName}" />
>>
>> <setHeader headerName="CamelAwsS3Key">
>> <simple>testMultiPart/${in.header.CamelFileName}
>> </simple>
>> </setHeader>
>>
>> <setHeader headerName="CamelAwsS3ContentLength">
>> <simple>${in.header.CamelFileLength}
>> </simple>
>> </setHeader>
>>
>> <log message="Send file to S3: ${properties:s3Endpoint}" />
>>
>> <to uri="aws-s3://{{s3Bucket}}?amazonS3Client=#amazonClient&multiPartUpload=true"
>> />
>>
>> <log message="Done sending file." />
>>
>> </route>
>> </camelContext>
>> </beans>
>>
>> Again, everything works fine if I set multiPartUpload=false in the above
>> camel-context.xml
>>
>> I have tried a lot of things like:
>>
>> • setting the CamelAwsS3ContentMD5 header to the MD5 hash of the
>> file (which doesn't make sense for multi-part files)
>> • various settings for the partSize parameter
>> • different sized files from very large to very small
>> • setting the system parameter: System.setProperty("com.
>> amazonaws.services.s3.disablePutObjectMD5Validation", "true");
>>
>> If I turn on trace debugging for camel, it doesnt help much:
>>
>> [d #2 - file://target/sendToS3/] S3Producer TRACE
>> Initiating multipart upload [com.amazonaws.services.s3.model.
>> InitiateMultipartUploadRequest@3731147a] from exchange
>> [Exchange[ID-Toucan-local-1515115111374-0-1]]...
>> [d #2 - file://target/sendToS3/] S3Producer TRACE
>> Uploading part [1] for testMultiPart/testfile.zip
>> [d #2 - file://target/sendToS3/] DefaultErrorHandler TRACE Is
>> exchangeId: ID-Toucan-local-1515115111374-0-1 interrupted? false
>> [d #2 - file://target/sendToS3/] DefaultErrorHandler TRACE Is
>> exchangeId: ID-Toucan-local-1515115111374-0-1 done? false
>> [d #2 - file://target/sendToS3/] DefaultErrorHandler TRACE
>> isRunAllowed() -> true (Run allowed if we are not stopped/stopping)
>> [d #2 - file://target/sendToS3/] DefaultExceptionPolicyStrategy TRACE
>> Finding best suited exception policy for thrown exception
>> com.amazonaws.services.s3.model.AmazonS3Exception
>> [d #2 - file://target/sendToS3/] DefaultExceptionPolicyStrategy TRACE
>> Finding best suited exception policy for thrown exception
>> com.amazonaws.services.s3.model.AmazonS3Exception
>> [d #2 - file://target/sendToS3/] DefaultExceptionPolicyStrategy TRACE
>> Found 0 candidates
>> [d #2 - file://target/sendToS3/] DefaultErrorHandler DEBUG
>> Failed delivery for (MessageId: ID-Toucan-local-1515115111374-0-2 on
>> ExchangeId: ID-Toucan-local-1515115111374-0-1). On delivery attempt: 0
>> caught: com.amazonaws.services.s3.model.AmazonS3Exception: null (Service:
>> Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID:
>> tx000000000000002e9edee-005a4ed3d2-2213a2-uky-campus-1; S3 Extended
>> Request ID: 2213a2-uky-campus-1-uky), S3 Extended Request ID:
>> 2213a2-uky-campus-1-uky
>> [d #2 - file://target/sendToS3/] DefaultErrorHandler TRACE
>> isRedeliveryAllowed() -> true (we are not stopping/stopped)
>> [d #2 - file://target/sendToS3/] DefaultErrorHandler TRACE This
>> exchange is not handled or continued so its marked as failed:
>> Exchange[ID-Toucan-local-1515115111374-0-1]
>> [d #2 - file://target/sendToS3/] InstrumentationProcessor TRACE to:
>> Recording duration: 356930 millis for exchange: Exchange[ID-Toucan-local-
>> 1515115111374-0-1]
>> [d #2 - file://target/sendToS3/] DefaultErrorHandler ERROR
>> Failed delivery for (MessageId: ID-Toucan-local-1515115111374-0-2 on
>> ExchangeId: ID-Toucan-local-1515115111374-0-1). Exhausted after delivery
>> attempt: 1 caught: com.amazonaws.services.s3.model.AmazonS3Exception:
>> null (Service: Amazon S3; Status Code: 403; Error Code:
>> SignatureDoesNotMatch; Request ID: tx000000000000002e9edee-
>> 005a4ed3d2-2213a2-uky-campus-1; S3 Extended Request ID:
>> 2213a2-uky-campus-1-uky), S3 Extended Request ID: 2213a2-uky-campus-1-uky
>>
>> The reason I need multi-part uploads to work is for very large files, but
>> I can't get it to work on any file size.
>>
>> (I posted this on StackOverflow since I am not sure how active this
>> mailing list is: https://stackoverflow.com/questions/48106180/apache-
>> camel-s3-upload-fails-with-status-code-403-signaturedoesnotmatch-when-mu <
>> https://stackoverflow.com/questions/48106180/apache-
>> camel-s3-upload-fails-with-status-code-403-signaturedoesnotmatch-when-mu>)
>>
>>
>>
Re: Problem with camel-aws S3 when multiPartUpload is set to true
Posted by Artur Jablonski <aj...@ravenpack.com>.
Try setting region on your AmazonS3Client bean.
On Fri, Jan 5, 2018 at 3:19 AM, kretin <kr...@mac.com> wrote:
> I created a simple camel route to poll for files in a local directory and
> upload them to a Ceph (S3) server at my University. I am using apache camel
> 2.20.0 with the camel-aws S3 component, when I set multiPartUpload=false
> (the default) in the uri, everything works fine, but if I change to
> multiPartUpload=true, it fails.
>
> I know there is nothing wrong with my s3 secret or s3 access key because
> when I set multiPartUpload=false, everything works (there are no crazy plus
> (+) characters that need to be escaped in the keys).
>
> Here is the stack trace:
>
> com.amazonaws.services.s3.model.AmazonS3Exception: null (Service: Amazon
> S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID:
> tx000000000000002e9edee-005a4ed3d2-2213a2-uky-campus-1; S3 Extended
> Request ID: 2213a2-uky-campus-1-uky)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.
> handleErrorResponse(AmazonHttpClient.java:1592)
> ~[aws-java-sdk-core-1.11.186.jar:?]
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.
> executeOneRequest(AmazonHttpClient.java:1257)
> ~[aws-java-sdk-core-1.11.186.jar:?]
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1029)
> ~[aws-java-sdk-core-1.11.186.jar:?]
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741)
> ~[aws-java-sdk-core-1.11.186.jar:?]
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.
> executeWithTimer(AmazonHttpClient.java:715) ~[aws-java-sdk-core-1.11.186.
> jar:?]
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697)
> ~[aws-java-sdk-core-1.11.186.jar:?]
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665)
> ~[aws-java-sdk-core-1.11.186.jar:?]
> at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.
> execute(AmazonHttpClient.java:647) ~[aws-java-sdk-core-1.11.186.jar:?]
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511)
> ~[aws-java-sdk-core-1.11.186.jar:?]
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4227)
> ~[aws-java-sdk-s3-1.11.186.jar:?]
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4174)
> ~[aws-java-sdk-s3-1.11.186.jar:?]
> at com.amazonaws.services.s3.AmazonS3Client.abortMultipartUpload(AmazonS3Client.java:2928)
> ~[aws-java-sdk-s3-1.11.186.jar:?]
> at org.apache.camel.component.aws.s3.S3Producer.
> processMultiPart(S3Producer.java:181) ~[camel-aws-2.20.0.jar:2.20.0]
> at org.apache.camel.component.aws.s3.S3Producer.process(S3Producer.java:84)
> ~[camel-aws-2.20.0.jar:2.20.0]
> at ...
>
> My camel-context.xml looks like:
>
> <?xml version="1.0" encoding="UTF-8"?>
> <beans xmlns="http://www.springframework.org/schema/beans"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="
> http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd
> http://camel.apache.org/schema/spring http://camel.apache.org/
> schema/spring/camel-spring.xsd">
>
> <bean id="properties"
> class="org.apache.camel.component.properties.PropertiesComponent">
> <property name="location" value="config.properties" />
> </bean>
>
> <bean id="bridgePropertyPlaceholder"
> class="org.apache.camel.spring.spi.BridgePropertyPlaceholderConfi
> gurer">
> <property name="systemPropertiesModeName"
> value="SYSTEM_PROPERTIES_MODE_OVERRIDE" />
> <property name="location" value="config.properties" />
> </bean>
>
> <bean id="amazonClient" class="com.amazonaws.services.
> s3.AmazonS3Client">
> <constructor-arg>
> <bean class="com.amazonaws.auth.BasicAWSCredentials">
> <constructor-arg name="accessKey" value="${s3AccessKey}" />
> <constructor-arg name="secretKey" value="${s3SecretKey}" />
> </bean>
> </constructor-arg>
> <property name="s3ClientOptions">
> <bean class="com.amazonaws.services.s3.S3ClientOptions">
> <property name="pathStyleAccess" value="true" />
> </bean>
> </property>
> <property name="endpoint" value="${s3Endpoint}" />
> </bean>
>
> <camelContext xmlns="http://camel.apache.org/schema/spring">
> <route>
> <from uri="file:target/sendToS3/?antInclude=*.*" />
>
> <log message="Found file: ${in.header.CamelFileName}" />
>
> <setHeader headerName="CamelAwsS3Key">
> <simple>testMultiPart/${in.header.CamelFileName}
> </simple>
> </setHeader>
>
> <setHeader headerName="CamelAwsS3ContentLength">
> <simple>${in.header.CamelFileLength}
> </simple>
> </setHeader>
>
> <log message="Send file to S3: ${properties:s3Endpoint}" />
>
> <to uri="aws-s3://{{s3Bucket}}?amazonS3Client=#amazonClient&multiPartUpload=true"
> />
>
> <log message="Done sending file." />
>
> </route>
> </camelContext>
> </beans>
>
> Again, everything works fine if I set multiPartUpload=false in the above
> camel-context.xml
>
> I have tried a lot of things like:
>
> • setting the CamelAwsS3ContentMD5 header to the MD5 hash of the
> file (which doesn't make sense for multi-part files)
> • various settings for the partSize parameter
> • different sized files from very large to very small
> • setting the system parameter: System.setProperty("com.
> amazonaws.services.s3.disablePutObjectMD5Validation", "true");
>
> If I turn on trace debugging for camel, it doesnt help much:
>
> [d #2 - file://target/sendToS3/] S3Producer TRACE
> Initiating multipart upload [com.amazonaws.services.s3.model.
> InitiateMultipartUploadRequest@3731147a] from exchange
> [Exchange[ID-Toucan-local-1515115111374-0-1]]...
> [d #2 - file://target/sendToS3/] S3Producer TRACE
> Uploading part [1] for testMultiPart/testfile.zip
> [d #2 - file://target/sendToS3/] DefaultErrorHandler TRACE Is
> exchangeId: ID-Toucan-local-1515115111374-0-1 interrupted? false
> [d #2 - file://target/sendToS3/] DefaultErrorHandler TRACE Is
> exchangeId: ID-Toucan-local-1515115111374-0-1 done? false
> [d #2 - file://target/sendToS3/] DefaultErrorHandler TRACE
> isRunAllowed() -> true (Run allowed if we are not stopped/stopping)
> [d #2 - file://target/sendToS3/] DefaultExceptionPolicyStrategy TRACE
> Finding best suited exception policy for thrown exception
> com.amazonaws.services.s3.model.AmazonS3Exception
> [d #2 - file://target/sendToS3/] DefaultExceptionPolicyStrategy TRACE
> Finding best suited exception policy for thrown exception
> com.amazonaws.services.s3.model.AmazonS3Exception
> [d #2 - file://target/sendToS3/] DefaultExceptionPolicyStrategy TRACE
> Found 0 candidates
> [d #2 - file://target/sendToS3/] DefaultErrorHandler DEBUG
> Failed delivery for (MessageId: ID-Toucan-local-1515115111374-0-2 on
> ExchangeId: ID-Toucan-local-1515115111374-0-1). On delivery attempt: 0
> caught: com.amazonaws.services.s3.model.AmazonS3Exception: null (Service:
> Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID:
> tx000000000000002e9edee-005a4ed3d2-2213a2-uky-campus-1; S3 Extended
> Request ID: 2213a2-uky-campus-1-uky), S3 Extended Request ID:
> 2213a2-uky-campus-1-uky
> [d #2 - file://target/sendToS3/] DefaultErrorHandler TRACE
> isRedeliveryAllowed() -> true (we are not stopping/stopped)
> [d #2 - file://target/sendToS3/] DefaultErrorHandler TRACE This
> exchange is not handled or continued so its marked as failed:
> Exchange[ID-Toucan-local-1515115111374-0-1]
> [d #2 - file://target/sendToS3/] InstrumentationProcessor TRACE to:
> Recording duration: 356930 millis for exchange: Exchange[ID-Toucan-local-
> 1515115111374-0-1]
> [d #2 - file://target/sendToS3/] DefaultErrorHandler ERROR
> Failed delivery for (MessageId: ID-Toucan-local-1515115111374-0-2 on
> ExchangeId: ID-Toucan-local-1515115111374-0-1). Exhausted after delivery
> attempt: 1 caught: com.amazonaws.services.s3.model.AmazonS3Exception:
> null (Service: Amazon S3; Status Code: 403; Error Code:
> SignatureDoesNotMatch; Request ID: tx000000000000002e9edee-
> 005a4ed3d2-2213a2-uky-campus-1; S3 Extended Request ID:
> 2213a2-uky-campus-1-uky), S3 Extended Request ID: 2213a2-uky-campus-1-uky
>
> The reason I need multi-part uploads to work is for very large files, but
> I can't get it to work on any file size.
>
> (I posted this on StackOverflow since I am not sure how active this
> mailing list is: https://stackoverflow.com/questions/48106180/apache-
> camel-s3-upload-fails-with-status-code-403-signaturedoesnotmatch-when-mu <
> https://stackoverflow.com/questions/48106180/apache-
> camel-s3-upload-fails-with-status-code-403-signaturedoesnotmatch-when-mu>)
>
>
>