You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/05/21 04:14:34 UTC
[jira] [Resolved] (SPARK-19189) Optimize CartesianRDD to avoid
parent RDD partition re-computation and re-serialization
[ https://issues.apache.org/jira/browse/SPARK-19189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-19189.
----------------------------------
Resolution: Incomplete
> Optimize CartesianRDD to avoid parent RDD partition re-computation and re-serialization
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-19189
> URL: https://issues.apache.org/jira/browse/SPARK-19189
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.1.0
> Reporter: Weichen Xu
> Assignee: Apache Spark
> Priority: Minor
> Labels: bulk-closed
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Current CartesianRDD implementation, suppose RDDA cartisian RDDB, generating RDDC,
> each RDDA partition will be reading by multiple RDDC partition, and RDDB has similar problem.
> This will cause, when RDDC partition computing, each partition's data in RDDA or RDDB will be repeatedly serialized (then transfer through network), if RDDA or RDDB haven't been persist, it will cause RDD recomputation repeatedly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org