You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "sam liu (JIRA)" <ji...@apache.org> on 2013/09/13 09:41:52 UTC
[jira] [Created] (MAPREDUCE-5506) Hadoop-1.1.1 occurs
ArrayIndexOutOfBoundsException with MultithreadedMapRunner
sam liu created MAPREDUCE-5506:
----------------------------------
Summary: Hadoop-1.1.1 occurs ArrayIndexOutOfBoundsException with MultithreadedMapRunner
Key: MAPREDUCE-5506
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5506
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1
Affects Versions: 1.1.1
Environment: RHEL 6.3 x86_64
Reporter: sam liu
Priority: Blocker
After I set:
- 'jobConf.setMapRunnerClass(MultithreadedMapRunner.class);' in MR app
- 'mapred.map.multithreadedrunner.threads = 2' in mapred-site.xml
A simple MR app failed as its Map task encountered ArrayIndexOutOfBoundsException as below(please ignore the line numbers in the exception as I added some log print codes):
java.lang.ArrayIndexOutOfBoundsException
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:1331)
at java.io.DataOutputStream.write(DataOutputStream.java:101)
at org.apache.hadoop.io.Text.write(Text.java:282)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1060)
at org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:591)
at study.hadoop.mapreduce.sample.WordCount$Map.map(WordCount.java:41)
at study.hadoop.mapreduce.sample.WordCount$Map.map(WordCount.java:1)
at org.apache.hadoop.mapred.lib.MultithreadedMapRunner$MapperInvokeRunable.run(MultithreadedMapRunner.java:231)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
at java.lang.Thread.run(Thread.java:738)
And the exception happens on line 'System.arraycopy(b, off, kvbuffer, bufindex, len)' in MapTask.java#MapOutputBuffer#Buffer#write(). When the exception occurs, 'b.length=4' but 'len=9'.
Btw, if I set 'mapred.map.multithreadedrunner.threads = 1', no exception happened. So it should be an issue caused by multiple threads.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira