Spark-submit and Windows / Linux mixed network

[Ashic Mahtab <as...@live.com>]

Hi,
I'm trying to submit a spark application fro network share to the spark master. Network shares are configured so that the master and all nodes have access to the target ja at (say):

\\shares\publish\Spark\app1\someJar.jar


And this is mounted on each linux box (i.e. master and workers) at:


/mnt/spark/app1/someJar.jar


I'm using the following to submit the app from a windows machine:


spark-submit.cmd --class Main --master spark://mastername:7077 local:/mnt/spark/app1/someJar.jar


However, I get an error saying:


Warning: Local jar \mnt\spark\app1\someJar.jar does not exist, skipping.


Followed by a ClassNotFoundException: Main.


Notice the \'s instead of /s. 


Does anybody have experience getting something similar to work?


Regards,
Ashic.

 		 	   		  

RE: Spark-submit and Windows / Linux mixed network

[Ashic Mahtab <as...@live.com>]

jar not found :(

Seems if I create a directory sim link so that the share path in the same on the unix mount point as in windows, and submit from the drive where the mount point is, then it works. Granted, that's quite an ugly hack.

Reverting to serving jar off http (i.e. using a relative path) for the time being.

Date: Tue, 11 Nov 2014 20:15:17 +0530
Subject: Re: Spark-submit and Windows / Linux mixed network
From: riteshoneinamillion@gmail.com
To: ashic@live.com
CC: user@spark.apache.org

Never tried this form but just guessing,
What's the output when you submit this jar: \\shares\publish\Spark\app1\someJar.jarusing spark-submit.cmd 		 	   		  

Re: Spark-submit and Windows / Linux mixed network

[Ritesh Kumar Singh <ri...@gmail.com>]

Never tried this form but just guessing,

What's the output when you submit this jar: \\shares\publish\Spark\app1\
someJar.jar
using spark-submit.cmd