[jira] [Commented] (SPARK-792) PairRDDFunctions should expect Product2 instead of Tuple2

["Sergei Lebedev (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16176268#comment-16176268 ] 

Sergei Lebedev commented on SPARK-792:
--------------------------------------

It looks like the change in the title was never implemented. Does anyone know why?

> PairRDDFunctions should expect Product2 instead of Tuple2
> ---------------------------------------------------------
>
>                 Key: SPARK-792
>                 URL: https://issues.apache.org/jira/browse/SPARK-792
>             Project: Spark
>          Issue Type: Improvement
>            Reporter: Reynold Xin
>            Assignee: Reynold Xin
>             Fix For: 0.8.0
>
>
> PairRDDFunctions expect an RDD of Tuple2 (key, value) pair. If we change it to expect Product2, we can apply for optimizations such as reducing the number of temporary objects allocated and a faster runtime.
> Product2 is just a trait in Scala, and users can define classes that extend Product2. Two use cases in particular I have in mind:
> 1. In MappedRDD or MappedPartitionRDD, the iterator could reuse a concrete Product2 object and always return that; but in every next() call, the iterator updates the content of the Product2.
> 2. In some Spark cases (such as Shark, GraphX), the key field is only used to partition the data. A concrete implementation of Product2 can have the key field (_1) marked as transient, and thus reducing the amount of shuffle data sent across the network between map and reduce stages.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org