[jira] [Resolved] (HBASE-13401) Snapshot export to S3 fails with jets3t error - The Content-MD5 you specified did not match what we received.

["Joseph Reid (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-13401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joseph Reid resolved HBASE-13401.
---------------------------------
    Resolution: Duplicate

Duplicate of [HBASE-13400|https://issues.apache.org/jira/browse/HBASE-13400].  My mistake.

> Snapshot export to S3 fails with jets3t error - The Content-MD5 you specified did not match what we received.
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-13401
>                 URL: https://issues.apache.org/jira/browse/HBASE-13401
>             Project: HBase
>          Issue Type: Bug
>          Components: hbase, mapreduce
>    Affects Versions: 0.98.0
>         Environment: CentOS 6.5, Hadoop 2.4.0, Yarn + MapReduce2 2.4.0, Hortonworks Data Platform 2.1.2
>            Reporter: Joseph Reid
>
> We're running into issues exporting snapshots of large tables to Amazon S3.
> The snapshot completes successfully, but the snapshot export job throws jets3t errors when we attempt to export to S3.  
> Error snippet, from job log:
> {code}
> 2015-04-03 16:59:16,425 INFO  [main] mapreduce.Job: Task Id : attempt_1426532296228_55454_m_000008_1, Status : FAILED
> Error: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 Error Message. -- ResponseCode: 400, ResponseStatus: Bad Request, XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>BadDigest</Code><Message>The Content-MD5 you specified did not match what we received.</Message><ExpectedDigest>CWiSsgzVAJyzPy2oT8u4Ag==</ExpectedDigest><CalculatedDigest>2DIsv6jZJ8FuGtalOO8SPA==</CalculatedDigest><RequestId>CA325C738970C313</RequestId><HostId>tnE+O1zPZovaQWMhCuM4lkX0h/wN9173FQ7omxZzLb6eH0OCHASyan+mb8WBJkNn</HostId></Error>
>   at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleS3ServiceException(Jets3tNativeFileSystemStore.java:405)
>   at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.storeFile(Jets3tNativeFileSystemStore.java:115)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>   at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>   at org.apache.hadoop.fs.s3native.$Proxy19.storeFile(Unknown Source)
>   at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.close(NativeS3FileSystem.java:221)
>   at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70)
>   at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:103)
>   at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.copyFile(ExportSnapshot.java:200)
>   at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:140)
>   at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:89)
>   at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
>   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
>   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
>   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: org.jets3t.service.S3ServiceException: S3 Error Message. -- ResponseCode: 400, ResponseStatus: Bad Request, XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>BadDigest</Code><Message>The Content-MD5 you specified did not match what we received.</Message><ExpectedDigest>CWiSsgzVAJyzPy2oT8u4Ag==</ExpectedDigest><CalculatedDigest>2DIsv6jZJ8FuGtalOO8SPA==</CalculatedDigest><RequestId>CA325C738970C313</RequestId><HostId>tnE+O1zPZovaQWMhCuM4lkX0h/wN9173FQ7omxZzLb6eH0OCHASyan+mb8WBJkNn</HostId></Error>
>   at org.jets3t.service.S3Service.putObject(S3Service.java:2267)
>   at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.storeFile(Jets3tNativeFileSystemStore.java:113)
>   ... 21 more
> 2015-04-03 17:03:50,613 INFO  [main] mapreduce.Job: Task Id : attempt_1426532296228_55454_m_000010_1, Status : FAILED
> AttemptID:attempt_1426532296228_55454_m_000010_1 Timed out after 300 secs
> {\code}
> We've verified that exports to other clusters from these same snapshots work fine.  Thus the issue appears to lie within the snapshot export utility, jets3t, and S3.
> "The Content-MD5 you specified did not match what we received" seems to indicate that the snapshot changed between when the upload started and the error.   Is that possible?  Why would a snapshot change?  
> Our jets3t.properties:
> {code}
> storage-service.internal-error-retry-max=5
> storage-service.disable-live-md5=false
> threaded-service.max-thread-count=20
> threaded-service.admin-max-thread-count=20
> s3service.max-thread-count=20
> s3service.admin-max-thread-count=20
> {\code}
> Related to:
> [Discussion on jets3t user group|https://groups.google.com/forum/#!topic/jets3t-users/Bg2qh7OdE2U]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)