You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by efwalkermit <ef...@alum.mit.edu> on 2016/01/19 00:34:47 UTC

PySpark Broadcast of User Defined Class No Work?

Should I be able to broadcast a fairly simple user-defined class?  I'm having
no success in 1.6.0 (or 1.5.2):

$ cat test_spark.py
import pyspark


class SimpleClass:
    def __init__(self):
        self.val = 5
    def get(self):
        return self.val


def main():
    sc = pyspark.SparkContext()
    b = sc.broadcast(SimpleClass())
    results = sc.parallelize([ x for x in range(10) ]).map(lambda x: x +
b.value.get()).collect() 

if __name__ == '__main__':
    main()


$ spark-submit --master local[1] test_spark.py
[snip]
  File "/Users/ed/src/mrspark/examples/fortyler/test_spark.py", line 14, in
<lambda>
    results = sc.parallelize([ x for x in range(10) ]).map(lambda x: x +
b.value.get()).collect()
  File
"/Users/ed/.spark/spark-1.5.2-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/broadcast.py",
line 97, in value
    self._value = self.load(self._path)
  File
"/Users/ed/.spark/spark-1.5.2-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/broadcast.py",
line 88, in load
    return pickle.load(f)
AttributeError: 'module' object has no attribute 'SimpleClass'
[snip]



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/PySpark-Broadcast-of-User-Defined-Class-No-Work-tp26000.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: PySpark Broadcast of User Defined Class No Work?

Posted by Maciej Szymkiewicz <ms...@gmail.com>.
Python can pickle only objects not classes. It means that SimpleClass
has to importable on every worker node to enable correct
deserialization. Typically it means keeping class definitions in a
separate module and distributing using for example --py-files.


On 01/19/2016 12:34 AM, efwalkermit wrote:
> Should I be able to broadcast a fairly simple user-defined class?  I'm having
> no success in 1.6.0 (or 1.5.2):
>
> $ cat test_spark.py
> import pyspark
>
>
> class SimpleClass:
>     def __init__(self):
>         self.val = 5
>     def get(self):
>         return self.val
>
>
> def main():
>     sc = pyspark.SparkContext()
>     b = sc.broadcast(SimpleClass())
>     results = sc.parallelize([ x for x in range(10) ]).map(lambda x: x +
> b.value.get()).collect() 
>
> if __name__ == '__main__':
>     main()
>
>
> $ spark-submit --master local[1] test_spark.py
> [snip]
>   File "/Users/ed/src/mrspark/examples/fortyler/test_spark.py", line 14, in
> <lambda>
>     results = sc.parallelize([ x for x in range(10) ]).map(lambda x: x +
> b.value.get()).collect()
>   File
> "/Users/ed/.spark/spark-1.5.2-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/broadcast.py",
> line 97, in value
>     self._value = self.load(self._path)
>   File
> "/Users/ed/.spark/spark-1.5.2-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/broadcast.py",
> line 88, in load
>     return pickle.load(f)
> AttributeError: 'module' object has no attribute 'SimpleClass'
> [snip]
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/PySpark-Broadcast-of-User-Defined-Class-No-Work-tp26000.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>