You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yin Huai (JIRA)" <ji...@apache.org> on 2015/04/17 20:56:59 UTC

[jira] [Commented] (SPARK-6986) Make SerializationStream/DeserializationStream understand key/value semantic

    [ https://issues.apache.org/jira/browse/SPARK-6986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500439#comment-14500439 ] 

Yin Huai commented on SPARK-6986:
---------------------------------

Seems the information we need to know is if an object is a key/value pair, a key, or a value. With this information, the underlying serializer can understand how to serialize an object (for a specialized serializer, it understands the structure of a key and a value). I am proposing the following interface changes

For {{SerializationStream}}, we add
{code}
def writeKey[T: ClassTag](key: T): SerializationStream
def writeValue[T: ClassTag](value: T): SerializationStream
{code}
When we want to serialize a key/value pair, we still use the general-purpose {{writeObject}} method.

For {{DeserializationStream}}, we add
{code}
def readKey[T: ClassTag](): T
def readValue[T: ClassTag](): T
{code}
When we want to deserialize a key/value pair, we still use the general-purpose {{readObject}} method.

For shuffle, since we do not directly interact with a {{SerializationStream}} and we use {{BlockObjectWriter}} write data out, we also need to add {{writeKey}} and {{writeValue}} to {{BlockObjectWriter}}. In {{ExternalSorter}}, {{spillToMergeableFile}} will use these two methods to write data out. For other places where we write key/value pairs, we still use the general-purpose {{write}} method of a {{BlockObjectWriter}}. 

[~matei] [~pwendell] [~rxin] [~marmbrus] What do you think?

> Make SerializationStream/DeserializationStream understand key/value semantic
> ----------------------------------------------------------------------------
>
>                 Key: SPARK-6986
>                 URL: https://issues.apache.org/jira/browse/SPARK-6986
>             Project: Spark
>          Issue Type: New Feature
>          Components: Spark Core, SQL
>            Reporter: Yin Huai
>            Priority: Blocker
>
> Our existing Java and Kryo serializer are both general-purpose serialize. They treat every object individually and encode the type of an object to underlying stream. For Spark, it is common that we serialize a collection with records having the same types (for example, records of a DataFrame). For these cases, we do not need to write out types of records and we can take advantage the type information to build specialized serializer. To do so, seems we need to extend the interface of SerializationStream/DeserializationStream, so a SerializationStream/DeserializationStream can have more information about objects passed in (for example, if an object is key/value pair, a key, or a value). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org