[jira] [Commented] (MAPREDUCE-2564) NullPointerException in WritableComparator

["Devaraj K (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13079969#comment-13079969 ] 

Devaraj K commented on MAPREDUCE-2564:
--------------------------------------

As per the code, we cannot use the WritableComparator directly because the constructors are made as protected. If there is no comparator for the output key class, it uses this way to create instance of it and it initializes the buffer and keys.

{code:xml}
  public static synchronized WritableComparator get(Class<? extends WritableComparable> c) {
    WritableComparator comparator = comparators.get(c);
    if (comparator == null)
      comparator = new WritableComparator(c, true);
    return comparator;
  }
{code}

When we are writing custom comparator and if we don’t want to use the existing compare() method we can use the below constructor from the subclass
{code:xml}
protected WritableComparator(Class<? extends WritableComparable> keyClass) {
    this(keyClass, false);
  }
{code}
Otherwise we can use this constructor
{code:xml}
protected WritableComparator(Class<? extends WritableComparable> keyClass,
      boolean createInstances)
{code}
Can you elaborate the usage scenario?


> NullPointerException in WritableComparator
> ------------------------------------------
>
>                 Key: MAPREDUCE-2564
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2564
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: task
>    Affects Versions: 0.20.203.0
>         Environment: java version "1.6.0_23"
> Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
> Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode)
> hadoop-0.20.203.0rc1
>            Reporter: Joseph Shraibman
>            Priority: Blocker
>
> java.lang.NullPointerException
>         at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:96)
>         at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1110)
>         at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:70)
>         at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
>         at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1398)
>         at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1297)
>         at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:698)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
>         at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210)
> It is easy to see why this is happening.  The WritableComparator is created in JobConf line 776:
> {code:title=JobConf.java}
>    return WritableComparator.get(getMapOutputKeyClass().asSubclass(WritableComparable.class));
> }
> {code}
> which calls
> {code:title=WritableComparator.java|borderStyle=solid}
> protected WritableComparator(Class<? extends WritableComparable> keyClass) {
>     this(keyClass, false);
>   }
>   protected WritableComparator(Class<? extends WritableComparable> keyClass,
>       boolean createInstances) {
>     this.keyClass = keyClass;
>     if (createInstances) {
>       key1 = newKey();
>       key2 = newKey();
>       buffer = new DataInputBuffer();
>     } else {
>       key1 = key2 = null;
>       buffer = null;
>     }
>   }
> {code}
> key1, key2, and buffer end up being null. When compare() is called the NPE is thrown because buffer is null
> {code:title=WritableComparator.java|borderStyle=solid}
>  public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
>     try {
>       buffer.reset(b1, s1, l1);                   // parse key1
>       key1.readFields(buffer);
>       
>       buffer.reset(b2, s2, l2);                   // parse key2
>       key2.readFields(buffer);
>       
>     } catch (IOException e) {
>       throw new RuntimeException(e);
>     }
>     
>     return compare(key1, key2);                   // compare them
>   }
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira