You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sebastian Straub (Jira)" <ji...@apache.org> on 2019/12/16 13:52:00 UTC
[jira] [Updated] (SPARK-30270) Can't pickle abstract classes (with
cloudpickle)
[ https://issues.apache.org/jira/browse/SPARK-30270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sebastian Straub updated SPARK-30270:
-------------------------------------
Description:
I can't use any classes that are derived from abstract classes in PySpark, because cloudpickle can't pickle them.
Example:
{code:java}
import pickle
from abc import ABC
from pyspark import cloudpickle
class Foo(ABC):
pass
class Bar(Foo):
pass
bar = Bar()
# pickle dump works fine
pickle.dumps(bar)
# cloudpickle doesn't
cloudpickle.dumps(bar)
{code}
A similar bug has already been reported in SPARK-21439 and marked resolved, but I can confirm that the issue still persists.
was:
I can't use any classes that are derived from abstract classes in PySpark, because cloudpickle can't pickle them. Example:
{code:java}
import pickle
from abc import ABC
from pyspark import cloudpickle
class Foo(ABC):
pass
class Bar(Foo):
pass
bar = Bar()
# pickle dump works fine
pickle.dumps(bar)
# cloudpickle doesn't
cloudpickle.dumps(bar)
{code}
A similar bug has already been reported in SPARK-21439 and marked resolved, but I can confirm that the issue still persists.
> Can't pickle abstract classes (with cloudpickle)
> ------------------------------------------------
>
> Key: SPARK-30270
> URL: https://issues.apache.org/jira/browse/SPARK-30270
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 2.4.4
> Reporter: Sebastian Straub
> Priority: Minor
> Labels: cloudpickle
>
> I can't use any classes that are derived from abstract classes in PySpark, because cloudpickle can't pickle them.
> Example:
> {code:java}
> import pickle
> from abc import ABC
> from pyspark import cloudpickle
> class Foo(ABC):
> pass
> class Bar(Foo):
> pass
> bar = Bar()
> # pickle dump works fine
> pickle.dumps(bar)
> # cloudpickle doesn't
> cloudpickle.dumps(bar)
> {code}
> A similar bug has already been reported in SPARK-21439 and marked resolved, but I can confirm that the issue still persists.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org