[jira] Created: (HADOOP-4243) Serialization framework use SequenceFile/TFile/Other metadata to instantiate deserializer

["Pete Wyckoff (JIRA)" <ji...@apache.org>]

Serialization framework use SequenceFile/TFile/Other metadata to instantiate deserializer
-----------------------------------------------------------------------------------------

                 Key: HADOOP-4243
                 URL: https://issues.apache.org/jira/browse/HADOOP-4243
             Project: Hadoop Core
          Issue Type: Improvement
          Components: contrib/serialization
            Reporter: Pete Wyckoff


SequenceFile metadata is useful for storing additional information about the serialized data, for example, for RecordIO, whether the data is CSV or Binary.  For thrift, the same thing - Binary, JSON, ...

For Hive, this may be especially important, because it has a Dynamic generic serializer/deserializer that takes its DDL at runtime (as opposed to RecordIO and Thrift which require pre-compilation into a specific class whose name can be stored in the sequence file key or value class).   In this case, the class name is like Record.java in RecordIO - it doesn't tell you anything without the DDL.

One way to address this could be adding the sequence file metadata to the getDeserializer call in Serialization interface.  The api would then be something like getDeserializer(Class<?>, Map<Text, Text> metadata) or Properties metadata.

But, I am open to proposals.

This also means that saying a class implements Writable is not enough to necessarily deserialize it since it may do specific actions based on the metadata - e.g., RecordIO might determine whether to use CSV rather than the default Binary deserialization.

There's the other issue of the getSerializer returning the metadata to be written to the Sequence/T File.



-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.