[jira] [Created] (MAPREDUCE-6581) Shuffle failure incase of NativeMapOutputCollectorDelegator with intermediate-data encrypt

["Bibin A Chundatt (JIRA)" <ji...@apache.org>]

Bibin A Chundatt created MAPREDUCE-6581:
-------------------------------------------

             Summary: Shuffle failure incase of NativeMapOutputCollectorDelegator with intermediate-data encrypt
                 Key: MAPREDUCE-6581
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6581
             Project: Hadoop Map/Reduce
          Issue Type: Bug
            Reporter: Bibin A Chundatt
            Priority: Blocker


*Steps to reproduce*

# Create data with teragen
# Run terasort on data prepared using teragen

Commands used 

./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar teragen 1024000 /Terainput1

./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar terasort -Dmapreduce.job.encrypted-intermediate-data=true -Dmapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator  -Dmapreduce.map.output.compress=true  -Dmapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.SnappyCodec -Dmapreduce.output.fileoutputformat.compress=true -Dmapreduce.output.fileoutputformat.compress.type=BLOCK -Dmapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.GzipCodec -Dmapreduce.reduce.memory.mb=1024 /Terainput1/Teraout12

{noformat}
15/12/18 23:07:57 INFO mapreduce.Job: Task Id : attempt_1450453391718_0017_r_000000_2, Status : FAILED
Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#5
	at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: java.lang.ArrayIndexOutOfBoundsException
	at org.apache.hadoop.io.compress.snappy.SnappyDecompressor.setInput(SnappyDecompressor.java:107)
	at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:104)
	at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
	at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:201)
	at org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.doShuffle(InMemoryMapOutput.java:90)
	at org.apache.hadoop.mapreduce.task.reduce.IFileWrappedMapOutput.shuffle(IFileWrappedMapOutput.java:63)
	at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:538)
	at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:336)
	at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)

{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)