You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Dhimant <dh...@gmail.com> on 2016/02/07 08:57:04 UTC
Re: Bad Digest error while doing aws s3 put
Hi , I am getting the following error while reading the huge data from S3 and
after processing ,writing data to S3 again.
Did you find any solution for this ?
16/02/07 07:41:59 WARN scheduler.TaskSetManager: Lost task 144.2 in stage
3.0 (TID 169, ip-172-31-7-26.us-west-2.compute.internal):
java.io.IOException: exception in uploadSinglePart
at
com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream.uploadSinglePart(MultipartUploadOutputStream.java:248)
at
com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream.close(MultipartUploadOutputStream.java:469)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:105)
at
org.apache.hadoop.io.compress.CompressorStream.close(CompressorStream.java:106)
at java.io.FilterOutputStream.close(FilterOutputStream.java:160)
at
org.apache.hadoop.mapred.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:109)
at
org.apache.spark.SparkHadoopWriter.close(SparkHadoopWriter.scala:102)
at
org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1080)
at
org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1059)
at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:64)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: exception in putObject
at
com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.storeFile(Jets3tNativeFileSystemStore.java:149)
at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
at com.sun.proxy.$Proxy26.storeFile(Unknown Source)
at
com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream.uploadSinglePart(MultipartUploadOutputStream.java:245)
... 15 more
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The
Content-MD5 you specified did not match what we received. (Service: Amazon
S3; Status Code: 400; Error Code: BadDigest; Request ID: 5918216A5901FCC8),
S3 Extended Request ID:
QSxtYln/yXqHYpdr4BWosin/TAFsGlK1FlKfE5PcuJkNrgoblGzTNt74kEhuNcrJCRZ3mXq0oUo=
at
com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)
at
com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770)
at
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3796)
at
com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1482)
at
com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.storeFile(Jets3tNativeFileSystemStore.java:140)
... 22 more
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Bad-Digest-error-while-doing-aws-s3-put-tp10036p26167.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Bad Digest error while doing aws s3 put
Posted by Eugen Cepoi <ce...@gmail.com>.
I had similar problems with multi part uploads. In my case the real error
was something else which was being masked by this issue
https://issues.apache.org/jira/browse/SPARK-6560. In the end this bad
digest exception was a side effect and not the original issue. For me it
was some library version conflict on EMR.
Depending on the size of the output files, you might try to just disable
the multipart upload using fs.s3n.multipart.uploads.enabled
Cheers,
Eugen
2016-02-07 15:05 GMT-08:00 Steve Loughran <st...@hortonworks.com>:
>
> > On 7 Feb 2016, at 07:57, Dhimant <dh...@gmail.com> wrote:
> >
> > at
> >
> com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream.uploadSinglePart(MultipartUploadOutputStream.java:245)
> > ... 15 more
> > Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The
> > Content-MD5 you specified did not match what we received. (Service:
> Amazon
> > S3; Status Code: 400; Error Code: BadDigest; Request ID:
> 5918216A5901FCC8),
> > S3 Extended Request ID:
> >
> QSxtYln/yXqHYpdr4BWosin/TAFsGlK1FlKfE5PcuJkNrgoblGzTNt74kEhuNcrJCRZ3mXq0oUo=
> > at
> >
> com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)
> > at
> >
> com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770)
> > at
> >
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
> > at
> > com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
> > at
> > com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3796)
> > at
> >
> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1482)
> > at
> >
> com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.storeFile(Jets3tNativeFileSystemStore.java:140)
> > ... 22 more
> >
>
> This is amazon's own s3 client. nothing in the apache hadoop source tree.
> Normally I'd say "use s3a to make s3n problems go away", but I don't know
> what that does on Amazon's own EMR libraries
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Bad Digest error while doing aws s3 put
Posted by Steve Loughran <st...@hortonworks.com>.
> On 7 Feb 2016, at 07:57, Dhimant <dh...@gmail.com> wrote:
>
> at
> com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream.uploadSinglePart(MultipartUploadOutputStream.java:245)
> ... 15 more
> Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The
> Content-MD5 you specified did not match what we received. (Service: Amazon
> S3; Status Code: 400; Error Code: BadDigest; Request ID: 5918216A5901FCC8),
> S3 Extended Request ID:
> QSxtYln/yXqHYpdr4BWosin/TAFsGlK1FlKfE5PcuJkNrgoblGzTNt74kEhuNcrJCRZ3mXq0oUo=
> at
> com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)
> at
> com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770)
> at
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3796)
> at
> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1482)
> at
> com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.storeFile(Jets3tNativeFileSystemStore.java:140)
> ... 22 more
>
This is amazon's own s3 client. nothing in the apache hadoop source tree. Normally I'd say "use s3a to make s3n problems go away", but I don't know what that does on Amazon's own EMR libraries
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Bad Digest error while doing aws s3 put
Posted by Steve Loughran <st...@hortonworks.com>.
> On 9 Feb 2016, at 07:19, lmk <la...@gmail.com> wrote:
>
> Hi Dhimant,
> As I had indicated in my next mail, my problem was due to disk getting full
> with log messages (these were dumped into the slaves) and did not have
> anything to do with the content pushed into s3. So, looks like this error
> message is very generic and is thrown for various reasons. You may probably
> have to do some more research to find out the cause of your problem..
> Please keep me posted once you fix this issue. Sorry, I could not be of much
> help to you..
>
> Regards
>
that's fun.
s3n/s3a buffer their output until close() is called, then they do a full upload
this breaks every assumption people have about file IO:
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/filesystem/introduction.html
-especially the bits in the code about close() being fast and harmless; now its O(data) and bad news if it fails.
If your close() was failing due to lack of HDD space, it means that your tmp dir and log dir were on the same disk/volume, and that ran out of capacity
HADOOP-11183 added an output variant which buffers in memory, primarily for faster output to rack-local storage supporting the s3 protocol. This is in ASF Hadoop 2.7, recent HDP and CDH releases.
I don't know if it's in amazon EMR, because they have their own closed source EMR client (believed to be a modified ASF one with some special hooks to unstable s3 APIs)
Anyway: I would run, not walk, to using s3a on Hadoop 2.7+, as its already better than s3a and getting better with every release
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Bad Digest error while doing aws s3 put
Posted by lmk <la...@gmail.com>.
Hi Dhimant,
As I had indicated in my next mail, my problem was due to disk getting full
with log messages (these were dumped into the slaves) and did not have
anything to do with the content pushed into s3. So, looks like this error
message is very generic and is thrown for various reasons. You may probably
have to do some more research to find out the cause of your problem..
Please keep me posted once you fix this issue. Sorry, I could not be of much
help to you..
Regards
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Bad-Digest-error-while-doing-aws-s3-put-tp10036p26174.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org