You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by "Rodriguez, John" <jr...@verisign.com> on 2010/09/14 21:16:05 UTC

Is deserialize(Writable field) able to handle any writable class

I have a custom Writable called DigestWritable. The DigestWritable is
stored in a SequenceFile.

 

For illustration, assume it is like this:

public class DigestWritable implements org.apache.hadoop.io.Writable {

  String str;

  int I;

}

 

I started to write a SerDe in order to read the DigestWritable from the
SequenceFile and pass the String/int fields to Hive.

 

However, in

  public Object deserialize(Writable field) throws SerDeException 

 

the Writable param seems to only be allowed to be Text or ByteWritable.

 

Is this correct, or am I doing something wrong in my SerDe?

 

How do I get the Writable to be a DigestWritable?

 

Is the Writable only allowed to be a primitive?

 

Thanks

John

 

 


RE: Is deserialize(Writable field) able to handle any writable class

Posted by "Rodriguez, John" <jr...@verisign.com>.
To answer my own question, the Writable can be any class that implements
Writable.

 

My "create external table" statement was not correct and I was not
getting the SequenceFile.reader which was causing my problems.

 

Also, even though the key in a SequenceFile is not used, it has to be
both Writable and WritableComparable. This is because of the method
signature from 

  src/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java

  private RecordReader<WritableComparable, Writable> getRecordReader()

 

In my SerDe, I now have access to my custom Writable class and I can
translate the values for Hive.

 

This is just an experiment with fewer fields than my custom Writable has
but here is the correct DDL.

 

create external table if not exists t10 (

    str string, flagWord int

)

partitioned by (id int)

row format 

  serde 'com.utils.DigestSerde'

STORED AS inputformat 

'org.apache.hadoop.mapred.SequenceFileInputFormat'

outputformat 

'org.apache.hadoop.mapred.SequenceFileOutputFormat'

location '/user/jrodrigu/hiveTables';

 

ALTER TABLE t10 ADD PARTITION (id=1) LOCATION
'/user/jrodrigu/hiveTables' PARTITION (id=2) LOCATION
'/user/jrodrigu/hiveTables1';

 

From: Rodriguez, John [mailto:jrodriguez@verisign.com] 
Sent: Tuesday, September 14, 2010 2:16 PM
To: hive-user@hadoop.apache.org
Subject: Is deserialize(Writable field) able to handle any writable
class

 

I have a custom Writable called DigestWritable. The DigestWritable is
stored in a SequenceFile.

 

For illustration, assume it is like this:

public class DigestWritable implements org.apache.hadoop.io.Writable {

  String str;

  int I;

}

 

I started to write a SerDe in order to read the DigestWritable from the
SequenceFile and pass the String/int fields to Hive.

 

However, in

  public Object deserialize(Writable field) throws SerDeException 

 

the Writable param seems to only be allowed to be Text or ByteWritable.

 

Is this correct, or am I doing something wrong in my SerDe?

 

How do I get the Writable to be a DigestWritable?

 

Is the Writable only allowed to be a primitive?

 

Thanks

John