Recommended way to develop spark application with both java and python

[Wush Wu <wu...@bridgewell.com>]

Dear all,

We have a spark 0.8.1 cluster on mesos 0.15. Some of my colleagues are
familiar with python, but some of features are developed under java. I am
looking for a way to integrate java and python on spark.

I notice that the initialization of pyspark does not include a field to
distribute jar files to slaves. After exploring the source code and do some
hacking, I could control the java sparkcontext object through py4j, but the
jar files are not delivered to slaves. Moreover, it seems that the spark
lauch the process through the spark home on pyspark but through the
spark.executor.uri on scala.

Is there a recommended way to develop spark application with both
java/scala and python? Should I suggest my team to unify the language?

Thanks!