You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Suman Sehgal (JIRA)" <ji...@apache.org> on 2009/07/08 07:44:14 UTC
[jira] Created: (HADOOP-6130) ArrayIndexOutOfBoundsException is
thrown by KeyFieldBasedPartitioner
ArrayIndexOutOfBoundsException is thrown by KeyFieldBasedPartitioner
--------------------------------------------------------------------
Key: HADOOP-6130
URL: https://issues.apache.org/jira/browse/HADOOP-6130
Project: Hadoop Common
Issue Type: Bug
Components: mapred
Affects Versions: 0.20.0
Reporter: Suman Sehgal
Priority: Critical
KeyFieldBasedPartitioner throws "KeyFieldBasedPartitioner" when some part of the specified key is missing.
Scenario :
=======
when value of num.key.fields.for.partition is greater than the separators provided in the input.
Command:
========
hadoop jar streaming.jar -Dmapred.reduce.tasks=3 -Dnum.key.fields.for.partition=5 -input <input-dir> -output <output-dir> -mapper org.apache.hadoop.mapred.lib.IdentityMapper -reducer org.apache.hadoop.mapred.lib.IdentityReducer -inputformat org.apache.hadoop.mapred.KeyValueTextInputFormat -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6130) ArrayIndexOutOfBoundsException is
thrown by KeyFieldBasedPartitioner
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-6130:
--------------------------------
Priority: Major (was: Critical)
Assignee: Amar Kamat
Lowering the priority since this bug would affect only those jobs that specify partitioning based on some key fields, and the generated keys doesn't have enough fields.
> ArrayIndexOutOfBoundsException is thrown by KeyFieldBasedPartitioner
> --------------------------------------------------------------------
>
> Key: HADOOP-6130
> URL: https://issues.apache.org/jira/browse/HADOOP-6130
> Project: Hadoop Common
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Suman Sehgal
> Assignee: Amar Kamat
>
> KeyFieldBasedPartitioner throws "KeyFieldBasedPartitioner" when some part of the specified key is missing.
> Scenario :
> =======
> when value of num.key.fields.for.partition is greater than the separators provided in the input.
> Command:
> ========
> hadoop jar streaming.jar -Dmapred.reduce.tasks=3 -Dnum.key.fields.for.partition=5 -input <input-dir> -output <output-dir> -mapper org.apache.hadoop.mapred.lib.IdentityMapper -reducer org.apache.hadoop.mapred.lib.IdentityReducer -inputformat org.apache.hadoop.mapred.KeyValueTextInputFormat -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6130) ArrayIndexOutOfBoundsException is
thrown by KeyFieldBasedPartitioner
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-6130:
-------------------------------
Attachment: HADOOP-6130-v1.0.patch
Attaching a patch the solves the issue which I could reproduce. Testing in progress.
> ArrayIndexOutOfBoundsException is thrown by KeyFieldBasedPartitioner
> --------------------------------------------------------------------
>
> Key: HADOOP-6130
> URL: https://issues.apache.org/jira/browse/HADOOP-6130
> Project: Hadoop Common
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Suman Sehgal
> Assignee: Amar Kamat
> Attachments: HADOOP-6130-v1.0.patch
>
>
> KeyFieldBasedPartitioner throws "KeyFieldBasedPartitioner" when some part of the specified key is missing.
> Scenario :
> =======
> when value of num.key.fields.for.partition is greater than the separators provided in the input.
> Command:
> ========
> hadoop jar streaming.jar -Dmapred.reduce.tasks=3 -Dnum.key.fields.for.partition=5 -input <input-dir> -output <output-dir> -mapper org.apache.hadoop.mapred.lib.IdentityMapper -reducer org.apache.hadoop.mapred.lib.IdentityReducer -inputformat org.apache.hadoop.mapred.KeyValueTextInputFormat -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6130) ArrayIndexOutOfBoundsException is
thrown by KeyFieldBasedPartitioner
Posted by "Suman Sehgal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728521#action_12728521 ]
Suman Sehgal commented on HADOOP-6130:
--------------------------------------
Stack trace:
=========
java.lang.ArrayIndexOutOfBoundsException: 42
at org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner.hashCode(KeyFieldBasedPartitioner.java:95)
at org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner.getPartition(KeyFieldBasedPartitioner.java:87)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:801)
at org.apache.hadoop.mapred.lib.IdentityMapper.map(IdentityMapper.java:40)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:356)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
> ArrayIndexOutOfBoundsException is thrown by KeyFieldBasedPartitioner
> --------------------------------------------------------------------
>
> Key: HADOOP-6130
> URL: https://issues.apache.org/jira/browse/HADOOP-6130
> Project: Hadoop Common
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Suman Sehgal
> Priority: Critical
>
> KeyFieldBasedPartitioner throws "KeyFieldBasedPartitioner" when some part of the specified key is missing.
> Scenario :
> =======
> when value of num.key.fields.for.partition is greater than the separators provided in the input.
> Command:
> ========
> hadoop jar streaming.jar -Dmapred.reduce.tasks=3 -Dnum.key.fields.for.partition=5 -input <input-dir> -output <output-dir> -mapper org.apache.hadoop.mapred.lib.IdentityMapper -reducer org.apache.hadoop.mapred.lib.IdentityReducer -inputformat org.apache.hadoop.mapred.KeyValueTextInputFormat -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.