You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by "Joseph Shraibman (JIRA)" <ji...@apache.org> on 2011/06/03 17:37:50 UTC
[jira] [Created] (MAPREDUCE-2564) NullPointerException in
WritableComparator
NullPointerException in WritableComparator
------------------------------------------
Key: MAPREDUCE-2564
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2564
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task
Affects Versions: 0.20.203.0
Environment: java version "1.6.0_23"
Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode)
hadoop-0.20.203.0rc1
Reporter: Joseph Shraibman
Priority: Blocker
java.lang.NullPointerException
at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:96)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1110)
at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:70)
at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1398)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1297)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:698)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210)
It is easy to see why this is happening. The WritableComparator is created in JobConf line 776:
{code:title=JobConf.java}
return WritableComparator.get(getMapOutputKeyClass().asSubclass(WritableComparable.class));
}
{code}
which calls
{code:title=WritableComparator.java|borderStyle=solid}
protected WritableComparator(Class<? extends WritableComparable> keyClass) {
this(keyClass, false);
}
protected WritableComparator(Class<? extends WritableComparable> keyClass,
boolean createInstances) {
this.keyClass = keyClass;
if (createInstances) {
key1 = newKey();
key2 = newKey();
buffer = new DataInputBuffer();
} else {
key1 = key2 = null;
buffer = null;
}
}
{code}
key1, key2, and buffer end up being null. When compare() is called the NPE is thrown because buffer is null
{code:title=WritableComparator.java|borderStyle=solid}
public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
try {
buffer.reset(b1, s1, l1); // parse key1
key1.readFields(buffer);
buffer.reset(b2, s2, l2); // parse key2
key2.readFields(buffer);
} catch (IOException e) {
throw new RuntimeException(e);
}
return compare(key1, key2); // compare them
}
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2564) NullPointerException in
WritableComparator
Posted by "Joseph Shraibman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joseph Shraibman resolved MAPREDUCE-2564.
-----------------------------------------
Resolution: Invalid
Fix Version/s: 0.20.203.0
> NullPointerException in WritableComparator
> ------------------------------------------
>
> Key: MAPREDUCE-2564
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2564
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task
> Affects Versions: 0.20.203.0
> Environment: java version "1.6.0_23"
> Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
> Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode)
> hadoop-0.20.203.0rc1
> Reporter: Joseph Shraibman
> Priority: Blocker
> Fix For: 0.20.203.0
>
>
> java.lang.NullPointerException
> at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:96)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1110)
> at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:70)
> at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1398)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1297)
> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:698)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210)
> It is easy to see why this is happening. The WritableComparator is created in JobConf line 776:
> {code:title=JobConf.java}
> return WritableComparator.get(getMapOutputKeyClass().asSubclass(WritableComparable.class));
> }
> {code}
> which calls
> {code:title=WritableComparator.java|borderStyle=solid}
> protected WritableComparator(Class<? extends WritableComparable> keyClass) {
> this(keyClass, false);
> }
> protected WritableComparator(Class<? extends WritableComparable> keyClass,
> boolean createInstances) {
> this.keyClass = keyClass;
> if (createInstances) {
> key1 = newKey();
> key2 = newKey();
> buffer = new DataInputBuffer();
> } else {
> key1 = key2 = null;
> buffer = null;
> }
> }
> {code}
> key1, key2, and buffer end up being null. When compare() is called the NPE is thrown because buffer is null
> {code:title=WritableComparator.java|borderStyle=solid}
> public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
> try {
> buffer.reset(b1, s1, l1); // parse key1
> key1.readFields(buffer);
>
> buffer.reset(b2, s2, l2); // parse key2
> key2.readFields(buffer);
>
> } catch (IOException e) {
> throw new RuntimeException(e);
> }
>
> return compare(key1, key2); // compare them
> }
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira