[jira] Commented: (HADOOP-120) Reading an ArrayWriter does not work because valueClass does not get initialized

["Owen O'Malley (JIRA)" <ji...@apache.org>]

    [ http://issues.apache.org/jira/browse/HADOOP-120?page=comments#action_12373224 ] 

Owen O'Malley commented on HADOOP-120:
--------------------------------------

After thinking about it for a bit, the problem with this patch is that this is going to encode the typename in each and every record. So if your value type is ArrayWriter<UTF8>, you are going to spend an extra 2+strlen("org.apache.hadoop.io.UTF8") bytes per a record. That's a fair amount of overhead.

We also have to be a careful with the serialization of ArrayWritable because it is used in the DFS name node logs.

I'm not sure what the right solution is. Probably for right now, I would derive a subclass of ArrayWritable that is specific for your type. It isn't pretty, but it is guaranteed to be safe.

public class UTF8Array extends ArrayWritable {
  public UTF8Array() {
     super(UTF8.class);
  }
}

> Reading an ArrayWriter does not work because valueClass does not get initialized
> --------------------------------------------------------------------------------
>
>          Key: HADOOP-120
>          URL: http://issues.apache.org/jira/browse/HADOOP-120
>      Project: Hadoop
>         Type: Bug

>   Components: io
>  Environment: Red Hat  
>     Reporter: Dick King
>  Attachments: hadoop-120-fix.patch
>
> If you have a Reducer whose value type is an ArrayWriter it gets enstreamed alright but at reconstruction type when ArrayWriter::readFields(DataInput in) runs on a DataInput that has a nonempty ArrayWriter , newInstance fails trying to instantiate the null class.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira