You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Joseph Shraibman (JIRA)" <ji...@apache.org> on 2011/06/03 17:37:50 UTC
[jira] [Created] (MAPREDUCE-2564) NullPointerException in
WritableComparator
NullPointerException in WritableComparator
------------------------------------------
Key: MAPREDUCE-2564
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2564
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task
Affects Versions: 0.20.203.0
Environment: java version "1.6.0_23"
Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode)
hadoop-0.20.203.0rc1
Reporter: Joseph Shraibman
Priority: Blocker
java.lang.NullPointerException
at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:96)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1110)
at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:70)
at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1398)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1297)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:698)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210)
It is easy to see why this is happening. The WritableComparator is created in JobConf line 776:
{code:title=JobConf.java}
return WritableComparator.get(getMapOutputKeyClass().asSubclass(WritableComparable.class));
}
{code}
which calls
{code:title=WritableComparator.java|borderStyle=solid}
protected WritableComparator(Class<? extends WritableComparable> keyClass) {
this(keyClass, false);
}
protected WritableComparator(Class<? extends WritableComparable> keyClass,
boolean createInstances) {
this.keyClass = keyClass;
if (createInstances) {
key1 = newKey();
key2 = newKey();
buffer = new DataInputBuffer();
} else {
key1 = key2 = null;
buffer = null;
}
}
{code}
key1, key2, and buffer end up being null. When compare() is called the NPE is thrown because buffer is null
{code:title=WritableComparator.java|borderStyle=solid}
public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
try {
buffer.reset(b1, s1, l1); // parse key1
key1.readFields(buffer);
buffer.reset(b2, s2, l2); // parse key2
key2.readFields(buffer);
} catch (IOException e) {
throw new RuntimeException(e);
}
return compare(key1, key2); // compare them
}
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2564) NullPointerException in
WritableComparator
Posted by "Joseph Shraibman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13079994#comment-13079994 ]
Joseph Shraibman commented on MAPREDUCE-2564:
---------------------------------------------
Hmm. This was fixed between 0.20.2 and 0.20.203. It seems I reported this against 203. Is it possible I used 0.20.2 and looked at the source of 0.20.2 while thinking it was 203? It is hard to remember what I was doing on that day two months ago.
> NullPointerException in WritableComparator
> ------------------------------------------
>
> Key: MAPREDUCE-2564
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2564
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task
> Affects Versions: 0.20.203.0
> Environment: java version "1.6.0_23"
> Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
> Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode)
> hadoop-0.20.203.0rc1
> Reporter: Joseph Shraibman
> Priority: Blocker
> Fix For: 0.20.203.0
>
>
> java.lang.NullPointerException
> at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:96)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1110)
> at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:70)
> at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1398)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1297)
> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:698)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210)
> It is easy to see why this is happening. The WritableComparator is created in JobConf line 776:
> {code:title=JobConf.java}
> return WritableComparator.get(getMapOutputKeyClass().asSubclass(WritableComparable.class));
> }
> {code}
> which calls
> {code:title=WritableComparator.java|borderStyle=solid}
> protected WritableComparator(Class<? extends WritableComparable> keyClass) {
> this(keyClass, false);
> }
> protected WritableComparator(Class<? extends WritableComparable> keyClass,
> boolean createInstances) {
> this.keyClass = keyClass;
> if (createInstances) {
> key1 = newKey();
> key2 = newKey();
> buffer = new DataInputBuffer();
> } else {
> key1 = key2 = null;
> buffer = null;
> }
> }
> {code}
> key1, key2, and buffer end up being null. When compare() is called the NPE is thrown because buffer is null
> {code:title=WritableComparator.java|borderStyle=solid}
> public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
> try {
> buffer.reset(b1, s1, l1); // parse key1
> key1.readFields(buffer);
>
> buffer.reset(b2, s2, l2); // parse key2
> key2.readFields(buffer);
>
> } catch (IOException e) {
> throw new RuntimeException(e);
> }
>
> return compare(key1, key2); // compare them
> }
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2564) NullPointerException in
WritableComparator
Posted by "Devaraj K (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13079969#comment-13079969 ]
Devaraj K commented on MAPREDUCE-2564:
--------------------------------------
As per the code, we cannot use the WritableComparator directly because the constructors are made as protected. If there is no comparator for the output key class, it uses this way to create instance of it and it initializes the buffer and keys.
{code:xml}
public static synchronized WritableComparator get(Class<? extends WritableComparable> c) {
WritableComparator comparator = comparators.get(c);
if (comparator == null)
comparator = new WritableComparator(c, true);
return comparator;
}
{code}
When we are writing custom comparator and if we don’t want to use the existing compare() method we can use the below constructor from the subclass
{code:xml}
protected WritableComparator(Class<? extends WritableComparable> keyClass) {
this(keyClass, false);
}
{code}
Otherwise we can use this constructor
{code:xml}
protected WritableComparator(Class<? extends WritableComparable> keyClass,
boolean createInstances)
{code}
Can you elaborate the usage scenario?
> NullPointerException in WritableComparator
> ------------------------------------------
>
> Key: MAPREDUCE-2564
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2564
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task
> Affects Versions: 0.20.203.0
> Environment: java version "1.6.0_23"
> Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
> Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode)
> hadoop-0.20.203.0rc1
> Reporter: Joseph Shraibman
> Priority: Blocker
>
> java.lang.NullPointerException
> at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:96)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1110)
> at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:70)
> at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1398)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1297)
> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:698)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210)
> It is easy to see why this is happening. The WritableComparator is created in JobConf line 776:
> {code:title=JobConf.java}
> return WritableComparator.get(getMapOutputKeyClass().asSubclass(WritableComparable.class));
> }
> {code}
> which calls
> {code:title=WritableComparator.java|borderStyle=solid}
> protected WritableComparator(Class<? extends WritableComparable> keyClass) {
> this(keyClass, false);
> }
> protected WritableComparator(Class<? extends WritableComparable> keyClass,
> boolean createInstances) {
> this.keyClass = keyClass;
> if (createInstances) {
> key1 = newKey();
> key2 = newKey();
> buffer = new DataInputBuffer();
> } else {
> key1 = key2 = null;
> buffer = null;
> }
> }
> {code}
> key1, key2, and buffer end up being null. When compare() is called the NPE is thrown because buffer is null
> {code:title=WritableComparator.java|borderStyle=solid}
> public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
> try {
> buffer.reset(b1, s1, l1); // parse key1
> key1.readFields(buffer);
>
> buffer.reset(b2, s2, l2); // parse key2
> key2.readFields(buffer);
>
> } catch (IOException e) {
> throw new RuntimeException(e);
> }
>
> return compare(key1, key2); // compare them
> }
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2564) NullPointerException in
WritableComparator
Posted by "Joseph Shraibman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joseph Shraibman resolved MAPREDUCE-2564.
-----------------------------------------
Resolution: Invalid
Fix Version/s: 0.20.203.0
> NullPointerException in WritableComparator
> ------------------------------------------
>
> Key: MAPREDUCE-2564
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2564
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task
> Affects Versions: 0.20.203.0
> Environment: java version "1.6.0_23"
> Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
> Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode)
> hadoop-0.20.203.0rc1
> Reporter: Joseph Shraibman
> Priority: Blocker
> Fix For: 0.20.203.0
>
>
> java.lang.NullPointerException
> at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:96)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1110)
> at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:70)
> at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1398)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1297)
> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:698)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210)
> It is easy to see why this is happening. The WritableComparator is created in JobConf line 776:
> {code:title=JobConf.java}
> return WritableComparator.get(getMapOutputKeyClass().asSubclass(WritableComparable.class));
> }
> {code}
> which calls
> {code:title=WritableComparator.java|borderStyle=solid}
> protected WritableComparator(Class<? extends WritableComparable> keyClass) {
> this(keyClass, false);
> }
> protected WritableComparator(Class<? extends WritableComparable> keyClass,
> boolean createInstances) {
> this.keyClass = keyClass;
> if (createInstances) {
> key1 = newKey();
> key2 = newKey();
> buffer = new DataInputBuffer();
> } else {
> key1 = key2 = null;
> buffer = null;
> }
> }
> {code}
> key1, key2, and buffer end up being null. When compare() is called the NPE is thrown because buffer is null
> {code:title=WritableComparator.java|borderStyle=solid}
> public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
> try {
> buffer.reset(b1, s1, l1); // parse key1
> key1.readFields(buffer);
>
> buffer.reset(b2, s2, l2); // parse key2
> key2.readFields(buffer);
>
> } catch (IOException e) {
> throw new RuntimeException(e);
> }
>
> return compare(key1, key2); // compare them
> }
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira