SPARK_SUBMIT_CLASSPATH Windows PYSPARK

[art1i <a....@reply.de>]

Hi,

So I was trying to get the Kafka Streaming working in a standalone python
application.

I needed to add the dependencies for this to work. The suggested way for
this is doing --jars using spark-submit, which is not practical considering
I wanted to launch and debug an application. Also you still have to set
-driver-class-path for this to work. 

So after searching around I found a JIRA, which said, that setting
SPARK_SUBMIT_CLASSPATH is workaround. This works on Linux, but on Windows
just setting the environmental var did not seem to work. So I looked into
spark-submit.cmd and it seems SPARK_SUBMIT_CLASSPATH is being set to:

set SPARK_SUBMIT_CLASSPATH= 

This is then overridden if  --driver-class-path is supplied. But  did not
want to supply the  argsument, I just wanted my enviromental variable to
persist like on Linux. So instead i changed the line

set SPARK_SUBMIT_CLASSPATH=%SPARK_SUBMIT_CLASSPATH%

Now everything works as expected and I can just inject the var in my python
script before importing  pypsark

os.environ["SPARK_SUBMIT_CLASSPATH"]="path1.jar;path2.jar"

So I was wondering if there is any reason this is not the default behavior
or I am just doing something wrong?

This is my first time posting, please excuse me if this is the wrong place
to post this.

Cheers,

Artyom
 
 



--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/SPARK-SUBMIT-CLASSPATH-Windows-PYSPARK-tp11868.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org