You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Surendranauth Hiraman <su...@velos.io> on 2014/07/01 20:31:38 UTC

PySpark Driver from Jython

Has anyone tried running pyspark driver code in Jython, preferably by
calling python code within Java code?

I know CPython is the only interpreter tested because of the need to
support C extensions.

But in my case, C extensions would be called on the worker, not in the
driver.

And being able to execute the python driver from within my JVM is an
advantage in my current use case.

-- 

SUREN HIRAMAN, VP TECHNOLOGY
Velos
Accelerating Machine Learning

440 NINTH AVENUE, 11TH FLOOR
NEW YORK, NY 10001
O: (917) 525-2466 ext. 105
F: 646.349.4063
E: suren.hiraman@v <su...@sociocast.com>elos.io
W: www.velos.io

Re: PySpark Driver from Jython

Posted by davies <da...@gmail.com>.

The function run in worker is serialized in driver, so the driver and worker
should be run in the same Python interpreter.

If you do not need c extension support, then Jython will be better than
CPython, because of the cost of serialization is much lower.

Davies



--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/PySpark-Driver-from-Jython-tp7142p7269.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.