You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Josh Rosen (JIRA)" <ji...@apache.org> on 2014/08/19 23:47:19 UTC
[jira] [Resolved] (SPARK-2790) PySpark zip() doesn't work properly
if RDDs have different serializers
[ https://issues.apache.org/jira/browse/SPARK-2790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Rosen resolved SPARK-2790.
-------------------------------
Resolution: Fixed
Fix Version/s: 1.1.0
> PySpark zip() doesn't work properly if RDDs have different serializers
> ----------------------------------------------------------------------
>
> Key: SPARK-2790
> URL: https://issues.apache.org/jira/browse/SPARK-2790
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 1.0.0, 1.1.0
> Reporter: Josh Rosen
> Assignee: Davies Liu
> Priority: Critical
> Fix For: 1.1.0
>
>
> In PySpark, attempting to {{zip()}} two RDDs may fail if the RDDs have different serializers (e.g. batched vs. unbatched), even if those RDDs have the same number of partitions and same numbers of elements. This problem occurs in the MLlib Python APIs, where we might want to zip a JavaRDD of LabelledPoints with a JavaRDD of batch-serialized Python objects.
> This is problematic because whether zip() succeeds or errors depends on the partitioning / batching strategy, and we don't want to surface the serialization details to users.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org