You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sebastian Straub (Jira)" <ji...@apache.org> on 2019/12/17 08:42:00 UTC
[jira] [Commented] (SPARK-30270) Can't pickle abstract classes
(with cloudpickle)
[ https://issues.apache.org/jira/browse/SPARK-30270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16997998#comment-16997998 ]
Sebastian Straub commented on SPARK-30270:
------------------------------------------
True, it's fixed on master, but not on branch-2.4 or any of the other 2.x branches. If I understand it correctly, stuff that's in the master branch will be released with Spark 3.0, but not necessarily with 2.x? As Spark 3 has not been released yet and 2.x will probably be around for a while, a backport would be highly appreciated.
> Can't pickle abstract classes (with cloudpickle)
> ------------------------------------------------
>
> Key: SPARK-30270
> URL: https://issues.apache.org/jira/browse/SPARK-30270
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 2.4.4
> Reporter: Sebastian Straub
> Priority: Minor
> Labels: cloudpickle
>
> I can't use any classes that are derived from abstract classes in PySpark, because cloudpickle can't pickle them.
> Example:
> {code:java}
> import pickle
> from abc import ABC
> from pyspark import cloudpickle
> class Foo(ABC):
> pass
> class Bar(Foo):
> pass
> bar = Bar()
> # pickle dump works fine
> pickle.dumps(bar)
> # cloudpickle doesn't
> cloudpickle.dumps(bar)
> {code}
> A similar bug has already been reported in SPARK-21439 and marked resolved, but I can confirm that the issue still persists.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org