You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2021/02/11 15:05:00 UTC
[jira] [Commented] (HADOOP-17500) S3A doesn't calculate Content-MD5
on uploads
[ https://issues.apache.org/jira/browse/HADOOP-17500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17283070#comment-17283070 ]
Steve Loughran commented on HADOOP-17500:
-----------------------------------------
ok, start with getting set up to run the hadoop integration test suites, as documented in: https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/testing.html
Then see what can be done with adding the building up of the MD5 data. Either it goes into the S3ABlockOutputStream or we put it into the buffers it builds up. Putting it in there (S3ADataBlocks) isolates better for unit testing.
> S3A doesn't calculate Content-MD5 on uploads
> --------------------------------------------
>
> Key: HADOOP-17500
> URL: https://issues.apache.org/jira/browse/HADOOP-17500
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Reporter: Pedro Tôrres
> Priority: Major
>
> Hadoop doesn't specify the Content-MD5 of an object when uploading it to an S3 Bucket. This prevents uploads to buckets with Object Lock, that require the Content-MD5 to be specified.
>
> {code:java}
> com.amazonaws.services.s3.model.AmazonS3Exception: Content-MD5 HTTP header is required for Put Part requests with Object Lock parameters (Service: Amazon S3; Status Code: 400; Error Code: InvalidRequest; Request ID: ****************; S3 Extended Request ID: ****************************************************************************; Proxy: null), S3 Extended Request ID: ****************************************************************************
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1819)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1403)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1372)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5248)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5195)
> at com.amazonaws.services.s3.AmazonS3Client.doUploadPart(AmazonS3Client.java:3768)
> at com.amazonaws.services.s3.AmazonS3Client.uploadPart(AmazonS3Client.java:3753)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.uploadPart(S3AFileSystem.java:2230)
> at org.apache.hadoop.fs.s3a.WriteOperationHelper.lambda$uploadPart$8(WriteOperationHelper.java:558)
> at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:110)
> ... 15 more{code}
>
> Similar to https://issues.apache.org/jira/browse/JCLOUDS-1549
> Related to https://issues.apache.org/jira/browse/HADOOP-13076
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org