[jira] Commented: (HADOOP-3787) Add serialization for Thrift

["Tom White (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-3787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12627937#action_12627937 ] 

Tom White commented on HADOOP-3787:
-----------------------------------

This, and HADOOP-1986 in general, does not mandate the use of SequenceFile. However, SequenceFiles are a convenient binary format, so that's what's I've used here for the example.

It would be possible to run MapReduce against Thrift records in flat files with a suitable InputFormat (which would need to be written), but such files would not be splittable (unless there is some general way to find Thrift record boundaries from an arbitrary position in the file). Unsplittable files do not in general play well with MapReduce and HDFS. Perhaps one way to fix this is to insert a special Thrift record every n records whose unique byte sequence can be scanned for to realign with the record boundaries. Could this work?

> Add serialization for Thrift
> ----------------------------
>
>                 Key: HADOOP-3787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3787
>             Project: Hadoop Core
>          Issue Type: Wish
>          Components: examples, mapred
>            Reporter: Tom White
>         Attachments: hadoop-3787.patch, libthrift.jar
>
>
> Thrift (http://incubator.apache.org/thrift/) is cross-language serialization and RPC framework. This issue is to write a ThriftSerialization to support using Thrift types in MapReduce programs, including an example program. This should probably go into contrib.
> (There is a prototype implementation in https://issues.apache.org/jira/secure/attachment/12370464/hadoop-serializer-v2.tar.gz)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.