You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Julien Beaudan <jb...@stottlerhenke.com> on 2015/07/17 23:40:55 UTC
Command builder problem when running worker in Windows
Hi,
I running a stand-alone cluster in Windows 7, and when I try to run any
worker on the machine, I get the following error:
15/07/17 14:14:43 ERROR ExecutorRunner: Error running executor
java.io.IOException: Cannot run program
"C:\cygdrive\c\Users\jbeaudan\Spark\spark-1.3.1-bin-hadoop2.4/bin/compute-classpath.cmd"
(in directory "."): CreateProcess error=2, The system cannot find the
file specified
at java.lang.ProcessBuilder.start(Unknown Source)
at org.apache.spark.util.Utils$.executeCommand(Utils.scala:1067)
at
org.apache.spark.util.Utils$.executeAndGetOutput(Utils.scala:1084)
at
org.apache.spark.deploy.worker.CommandUtils$.buildJavaOpts(CommandUtils.scala:112)
at
org.apache.spark.deploy.worker.CommandUtils$.buildCommandSeq(CommandUtils.scala:61)
at
org.apache.spark.deploy.worker.CommandUtils$.buildProcessBuilder(CommandUtils.scala:47)
at
org.apache.spark.deploy.worker.ExecutorRunner.fetchAndRunExecutor(ExecutorRunner.scala:132)
at
org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:68)
Caused by: java.io.IOException: CreateProcess error=2, The system cannot
find the file specified
at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.<init>(Unknown Source)
at java.lang.ProcessImpl.start(Unknown Source)
... 8 more
I'm pretty sure the problem is that Spark is looking for the following
path, which mixes forward and back slashes:
C:\cygdrive\c\Users\jbeaudan\Spark\spark-1.3.1-bin-hadoop2.4/bin/compute-classpath.cmd
Is there anyway to fix this?
(Also, I have also tried running this from a normal terminal, instead of
from cygwin, and I get the same issue, except this time the path is:
C:\Users\jbeaudan\Spark\spark-1.3.1-bin-hadoop2.4\bin../bin/compute-classpath.cmd
)
Thank you!
Julien
Re: Command builder problem when running worker in Windows
Posted by Elkhan Dadashov <el...@gmail.com>.
My guess, you need to check 2 things:
1) sparkTest1.jar contains tests.testFileReader
2) put sparkTest1.jar into the directory from where you are executing this
command, and then run this :
bin/spark-submit --verbose --class tests.testFileReader --master spark://
192.168.194.128:7077 sparkTest1.jar
This is another working command for running SparkPi example which comes
with Spark:
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master
yarn-cluster lib/spark-examples*.jar 10
Hope it helps.
On Fri, Jul 17, 2015 at 3:44 PM, Julien Beaudan <jb...@stottlerhenke.com>
wrote:
> Oh, yeah of course. I'm writing from the command line (I haven't tried
> the SparkLauncher), using
>
> bin/spark-submit --class tests.testFileReader --master spark://
> 192.168.194.128:7077 --verbose ./sparkTest1.jar
>
> All that the testFileReader class does is create an RDD from a few text
> files - just a sanity check to make sure that my set up works.
>
> Julien
>
>
> On 07/17/2015 03:35 PM, Elkhan Dadashov wrote:
>
> Are you running it from command line (CLI) or through SparkLauncher ?
>
> If you can share the command (./bin/spark-submit ...) or the code
> snippet you are running, then it can give some clue.
>
> On Fri, Jul 17, 2015 at 3:30 PM, Julien Beaudan <
> jbeaudan@stottlerhenke.com> wrote:
>
>> Hi Elkhan,
>>
>> I ran Spark with --verbose, but the output looked the same to me - what
>> should I be looking for? At the beginning, the system properties which are
>> set are:
>>
>> System properties:
>> SPARK_SUBMIT -> true
>> spark.app.name -> tests.testFileReader
>> spark.jars ->
>> file:/C:/Users/jbeaudan/Spark/spark-1.3.1-bin-hadoop2.4/sparkTest1.jar
>> spark.master -> spark://192.168.194.128:7077
>> Classpath elements:
>> file:/C:/Users/jbeaudan/Spark/spark-1.3.1-bin-hadoop2.4/sparkTest1.jar
>>
>> I'm not sure why, but the file paths here seem formatted correctly (it is
>> same from the command terminal and Cygwin), so the path must get edited
>> afterwards?
>>
>> Julien
>>
>>
>> On 07/17/2015 03:00 PM, Elkhan Dadashov wrote:
>>
>> Run Spark with --verbose flag, to see what it read for that path.
>>
>> I guess in Windows if you are using backslash, you need 2 of them (\\),
>> or just use forward slashes everywhere.
>>
>> On Fri, Jul 17, 2015 at 2:40 PM, Julien Beaudan <
>> jbeaudan@stottlerhenke.com> wrote:
>>
>>> Hi,
>>>
>>> I running a stand-alone cluster in Windows 7, and when I try to run any
>>> worker on the machine, I get the following error:
>>>
>>> 15/07/17 14:14:43 ERROR ExecutorRunner: Error running executor
>>> java.io.IOException: Cannot run program
>>> "C:\cygdrive\c\Users\jbeaudan\Spark\spark-1.3.1-bin-hadoop2.4/bin/compute-classpath.cmd"
>>> (in directory "."): CreateProcess error=2, The system cannot find the file
>>> specified
>>> at java.lang.ProcessBuilder.start(Unknown Source)
>>> at org.apache.spark.util.Utils$.executeCommand(Utils.scala:1067)
>>> at
>>> org.apache.spark.util.Utils$.executeAndGetOutput(Utils.scala:1084)
>>> at
>>> org.apache.spark.deploy.worker.CommandUtils$.buildJavaOpts(CommandUtils.scala:112)
>>> at
>>> org.apache.spark.deploy.worker.CommandUtils$.buildCommandSeq(CommandUtils.scala:61)
>>> at
>>> org.apache.spark.deploy.worker.CommandUtils$.buildProcessBuilder(CommandUtils.scala:47)
>>> at
>>> org.apache.spark.deploy.worker.ExecutorRunner.fetchAndRunExecutor(ExecutorRunner.scala:132)
>>> at
>>> org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:68)
>>> Caused by: java.io.IOException: CreateProcess error=2, The system cannot
>>> find the file specified
>>> at java.lang.ProcessImpl.create(Native Method)
>>> at java.lang.ProcessImpl.<init>(Unknown Source)
>>> at java.lang.ProcessImpl.start(Unknown Source)
>>> ... 8 more
>>>
>>>
>>> I'm pretty sure the problem is that Spark is looking for the following
>>> path, which mixes forward and back slashes:
>>>
>>>
>>> C:\cygdrive\c\Users\jbeaudan\Spark\spark-1.3.1-bin-hadoop2.4/bin/compute-classpath.cmd
>>>
>>> Is there anyway to fix this?
>>>
>>> (Also, I have also tried running this from a normal terminal, instead of
>>> from cygwin, and I get the same issue, except this time the path is:
>>> C:\Users\jbeaudan\Spark\spark-1.3.1-bin-hadoop2.4\bin../bin/compute-classpath.cmd
>>> )
>>>
>>> Thank you!
>>>
>>> Julien
>>>
>>>
>>>
>>>
>>
>>
>> --
>>
>> Best regards,
>> Elkhan Dadashov
>>
>>
>>
>
>
> --
>
> Best regards,
> Elkhan Dadashov
>
>
>
--
Best regards,
Elkhan Dadashov
Re: Command builder problem when running worker in Windows
Posted by Julien Beaudan <jb...@stottlerhenke.com>.
Oh, yeah of course. I'm writing from the command line (I haven't tried
the SparkLauncher), using
bin/spark-submit --class tests.testFileReader --master
spark://192.168.194.128:7077 --verbose ./sparkTest1.jar
All that the testFileReader class does is create an RDD from a few text
files - just a sanity check to make sure that my set up works.
Julien
On 07/17/2015 03:35 PM, Elkhan Dadashov wrote:
> Are you running it from command line (CLI) or through SparkLauncher ?
>
> If you can share the command (./bin/spark-submit ...) or the code
> snippet you are running, then it can give some clue.
>
> On Fri, Jul 17, 2015 at 3:30 PM, Julien Beaudan
> <jbeaudan@stottlerhenke.com <ma...@stottlerhenke.com>> wrote:
>
> Hi Elkhan,
>
> I ran Spark with --verbose, but the output looked the same to me -
> what should I be looking for? At the beginning, the system
> properties which are set are:
>
> System properties:
> SPARK_SUBMIT -> true
> spark.app.name <http://spark.app.name> -> tests.testFileReader
> spark.jars ->
> file:/C:/Users/jbeaudan/Spark/spark-1.3.1-bin-hadoop2.4/sparkTest1.jar
> spark.master -> spark://192.168.194.128:7077
> <http://192.168.194.128:7077>
> Classpath elements:
> file:/C:/Users/jbeaudan/Spark/spark-1.3.1-bin-hadoop2.4/sparkTest1.jar
>
> I'm not sure why, but the file paths here seem formatted correctly
> (it is same from the command terminal and Cygwin), so the path
> must get edited afterwards?
>
> Julien
>
>
> On 07/17/2015 03:00 PM, Elkhan Dadashov wrote:
>> Run Spark with --verbose flag, to see what it read for that path.
>>
>> I guess in Windows if you are using backslash, you need 2 of them
>> (\\), or just use forward slashes everywhere.
>>
>> On Fri, Jul 17, 2015 at 2:40 PM, Julien Beaudan
>> <jbeaudan@stottlerhenke.com <ma...@stottlerhenke.com>>
>> wrote:
>>
>> Hi,
>>
>> I running a stand-alone cluster in Windows 7, and when I try
>> to run any worker on the machine, I get the following error:
>>
>> 15/07/17 14:14:43 ERROR ExecutorRunner: Error running executor
>> java.io.IOException: Cannot run program
>> "C:\cygdrive\c\Users\jbeaudan\Spark\spark-1.3.1-bin-hadoop2.4/bin/compute-classpath.cmd"
>> (in directory "."): CreateProcess error=2, The system cannot
>> find the file specified
>> at java.lang.ProcessBuilder.start(Unknown Source)
>> at
>> org.apache.spark.util.Utils$.executeCommand(Utils.scala:1067)
>> at
>> org.apache.spark.util.Utils$.executeAndGetOutput(Utils.scala:1084)
>> at
>> org.apache.spark.deploy.worker.CommandUtils$.buildJavaOpts(CommandUtils.scala:112)
>> at
>> org.apache.spark.deploy.worker.CommandUtils$.buildCommandSeq(CommandUtils.scala:61)
>> at
>> org.apache.spark.deploy.worker.CommandUtils$.buildProcessBuilder(CommandUtils.scala:47)
>> at
>> org.apache.spark.deploy.worker.ExecutorRunner.fetchAndRunExecutor(ExecutorRunner.scala:132)
>> at
>> org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:68)
>> Caused by: java.io.IOException: CreateProcess error=2, The
>> system cannot find the file specified
>> at java.lang.ProcessImpl.create(Native Method)
>> at java.lang.ProcessImpl.<init>(Unknown Source)
>> at java.lang.ProcessImpl.start(Unknown Source)
>> ... 8 more
>>
>>
>> I'm pretty sure the problem is that Spark is looking for the
>> following path, which mixes forward and back slashes:
>>
>> C:\cygdrive\c\Users\jbeaudan\Spark\spark-1.3.1-bin-hadoop2.4/bin/compute-classpath.cmd
>>
>> Is there anyway to fix this?
>>
>> (Also, I have also tried running this from a normal terminal,
>> instead of from cygwin, and I get the same issue, except this
>> time the path is:
>> C:\Users\jbeaudan\Spark\spark-1.3.1-bin-hadoop2.4\bin../bin/compute-classpath.cmd
>> )
>>
>> Thank you!
>>
>> Julien
>>
>>
>>
>>
>>
>>
>> --
>>
>> Best regards,
>> Elkhan Dadashov
>
>
>
>
> --
>
> Best regards,
> Elkhan Dadashov
Re: Command builder problem when running worker in Windows
Posted by Elkhan Dadashov <el...@gmail.com>.
Are you running it from command line (CLI) or through SparkLauncher ?
If you can share the command (./bin/spark-submit ...) or the code snippet
you are running, then it can give some clue.
On Fri, Jul 17, 2015 at 3:30 PM, Julien Beaudan <jb...@stottlerhenke.com>
wrote:
> Hi Elkhan,
>
> I ran Spark with --verbose, but the output looked the same to me - what
> should I be looking for? At the beginning, the system properties which are
> set are:
>
> System properties:
> SPARK_SUBMIT -> true
> spark.app.name -> tests.testFileReader
> spark.jars ->
> file:/C:/Users/jbeaudan/Spark/spark-1.3.1-bin-hadoop2.4/sparkTest1.jar
> spark.master -> spark://192.168.194.128:7077
> Classpath elements:
> file:/C:/Users/jbeaudan/Spark/spark-1.3.1-bin-hadoop2.4/sparkTest1.jar
>
> I'm not sure why, but the file paths here seem formatted correctly (it is
> same from the command terminal and Cygwin), so the path must get edited
> afterwards?
>
> Julien
>
>
> On 07/17/2015 03:00 PM, Elkhan Dadashov wrote:
>
> Run Spark with --verbose flag, to see what it read for that path.
>
> I guess in Windows if you are using backslash, you need 2 of them (\\),
> or just use forward slashes everywhere.
>
> On Fri, Jul 17, 2015 at 2:40 PM, Julien Beaudan <
> jbeaudan@stottlerhenke.com> wrote:
>
>> Hi,
>>
>> I running a stand-alone cluster in Windows 7, and when I try to run any
>> worker on the machine, I get the following error:
>>
>> 15/07/17 14:14:43 ERROR ExecutorRunner: Error running executor
>> java.io.IOException: Cannot run program
>> "C:\cygdrive\c\Users\jbeaudan\Spark\spark-1.3.1-bin-hadoop2.4/bin/compute-classpath.cmd"
>> (in directory "."): CreateProcess error=2, The system cannot find the file
>> specified
>> at java.lang.ProcessBuilder.start(Unknown Source)
>> at org.apache.spark.util.Utils$.executeCommand(Utils.scala:1067)
>> at
>> org.apache.spark.util.Utils$.executeAndGetOutput(Utils.scala:1084)
>> at
>> org.apache.spark.deploy.worker.CommandUtils$.buildJavaOpts(CommandUtils.scala:112)
>> at
>> org.apache.spark.deploy.worker.CommandUtils$.buildCommandSeq(CommandUtils.scala:61)
>> at
>> org.apache.spark.deploy.worker.CommandUtils$.buildProcessBuilder(CommandUtils.scala:47)
>> at
>> org.apache.spark.deploy.worker.ExecutorRunner.fetchAndRunExecutor(ExecutorRunner.scala:132)
>> at
>> org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:68)
>> Caused by: java.io.IOException: CreateProcess error=2, The system cannot
>> find the file specified
>> at java.lang.ProcessImpl.create(Native Method)
>> at java.lang.ProcessImpl.<init>(Unknown Source)
>> at java.lang.ProcessImpl.start(Unknown Source)
>> ... 8 more
>>
>>
>> I'm pretty sure the problem is that Spark is looking for the following
>> path, which mixes forward and back slashes:
>>
>>
>> C:\cygdrive\c\Users\jbeaudan\Spark\spark-1.3.1-bin-hadoop2.4/bin/compute-classpath.cmd
>>
>> Is there anyway to fix this?
>>
>> (Also, I have also tried running this from a normal terminal, instead of
>> from cygwin, and I get the same issue, except this time the path is:
>> C:\Users\jbeaudan\Spark\spark-1.3.1-bin-hadoop2.4\bin../bin/compute-classpath.cmd
>> )
>>
>> Thank you!
>>
>> Julien
>>
>>
>>
>>
>
>
> --
>
> Best regards,
> Elkhan Dadashov
>
>
>
--
Best regards,
Elkhan Dadashov
Re: Command builder problem when running worker in Windows
Posted by Julien Beaudan <jb...@stottlerhenke.com>.
Hi Elkhan,
I ran Spark with --verbose, but the output looked the same to me - what
should I be looking for? At the beginning, the system properties which
are set are:
System properties:
SPARK_SUBMIT -> true
spark.app.name -> tests.testFileReader
spark.jars ->
file:/C:/Users/jbeaudan/Spark/spark-1.3.1-bin-hadoop2.4/sparkTest1.jar
spark.master -> spark://192.168.194.128:7077
Classpath elements:
file:/C:/Users/jbeaudan/Spark/spark-1.3.1-bin-hadoop2.4/sparkTest1.jar
I'm not sure why, but the file paths here seem formatted correctly (it
is same from the command terminal and Cygwin), so the path must get
edited afterwards?
Julien
On 07/17/2015 03:00 PM, Elkhan Dadashov wrote:
> Run Spark with --verbose flag, to see what it read for that path.
>
> I guess in Windows if you are using backslash, you need 2 of them
> (\\), or just use forward slashes everywhere.
>
> On Fri, Jul 17, 2015 at 2:40 PM, Julien Beaudan
> <jbeaudan@stottlerhenke.com <ma...@stottlerhenke.com>> wrote:
>
> Hi,
>
> I running a stand-alone cluster in Windows 7, and when I try to
> run any worker on the machine, I get the following error:
>
> 15/07/17 14:14:43 ERROR ExecutorRunner: Error running executor
> java.io.IOException: Cannot run program
> "C:\cygdrive\c\Users\jbeaudan\Spark\spark-1.3.1-bin-hadoop2.4/bin/compute-classpath.cmd"
> (in directory "."): CreateProcess error=2, The system cannot find
> the file specified
> at java.lang.ProcessBuilder.start(Unknown Source)
> at
> org.apache.spark.util.Utils$.executeCommand(Utils.scala:1067)
> at
> org.apache.spark.util.Utils$.executeAndGetOutput(Utils.scala:1084)
> at
> org.apache.spark.deploy.worker.CommandUtils$.buildJavaOpts(CommandUtils.scala:112)
> at
> org.apache.spark.deploy.worker.CommandUtils$.buildCommandSeq(CommandUtils.scala:61)
> at
> org.apache.spark.deploy.worker.CommandUtils$.buildProcessBuilder(CommandUtils.scala:47)
> at
> org.apache.spark.deploy.worker.ExecutorRunner.fetchAndRunExecutor(ExecutorRunner.scala:132)
> at
> org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:68)
> Caused by: java.io.IOException: CreateProcess error=2, The system
> cannot find the file specified
> at java.lang.ProcessImpl.create(Native Method)
> at java.lang.ProcessImpl.<init>(Unknown Source)
> at java.lang.ProcessImpl.start(Unknown Source)
> ... 8 more
>
>
> I'm pretty sure the problem is that Spark is looking for the
> following path, which mixes forward and back slashes:
>
> C:\cygdrive\c\Users\jbeaudan\Spark\spark-1.3.1-bin-hadoop2.4/bin/compute-classpath.cmd
>
> Is there anyway to fix this?
>
> (Also, I have also tried running this from a normal terminal,
> instead of from cygwin, and I get the same issue, except this time
> the path is:
> C:\Users\jbeaudan\Spark\spark-1.3.1-bin-hadoop2.4\bin../bin/compute-classpath.cmd
> )
>
> Thank you!
>
> Julien
>
>
>
>
>
>
> --
>
> Best regards,
> Elkhan Dadashov
Re: Command builder problem when running worker in Windows
Posted by Elkhan Dadashov <el...@gmail.com>.
Run Spark with --verbose flag, to see what it read for that path.
I guess in Windows if you are using backslash, you need 2 of them (\\), or
just use forward slashes everywhere.
On Fri, Jul 17, 2015 at 2:40 PM, Julien Beaudan <jb...@stottlerhenke.com>
wrote:
> Hi,
>
> I running a stand-alone cluster in Windows 7, and when I try to run any
> worker on the machine, I get the following error:
>
> 15/07/17 14:14:43 ERROR ExecutorRunner: Error running executor
> java.io.IOException: Cannot run program
> "C:\cygdrive\c\Users\jbeaudan\Spark\spark-1.3.1-bin-hadoop2.4/bin/compute-classpath.cmd"
> (in directory "."): CreateProcess error=2, The system cannot find the file
> specified
> at java.lang.ProcessBuilder.start(Unknown Source)
> at org.apache.spark.util.Utils$.executeCommand(Utils.scala:1067)
> at
> org.apache.spark.util.Utils$.executeAndGetOutput(Utils.scala:1084)
> at
> org.apache.spark.deploy.worker.CommandUtils$.buildJavaOpts(CommandUtils.scala:112)
> at
> org.apache.spark.deploy.worker.CommandUtils$.buildCommandSeq(CommandUtils.scala:61)
> at
> org.apache.spark.deploy.worker.CommandUtils$.buildProcessBuilder(CommandUtils.scala:47)
> at
> org.apache.spark.deploy.worker.ExecutorRunner.fetchAndRunExecutor(ExecutorRunner.scala:132)
> at
> org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:68)
> Caused by: java.io.IOException: CreateProcess error=2, The system cannot
> find the file specified
> at java.lang.ProcessImpl.create(Native Method)
> at java.lang.ProcessImpl.<init>(Unknown Source)
> at java.lang.ProcessImpl.start(Unknown Source)
> ... 8 more
>
>
> I'm pretty sure the problem is that Spark is looking for the following
> path, which mixes forward and back slashes:
>
>
> C:\cygdrive\c\Users\jbeaudan\Spark\spark-1.3.1-bin-hadoop2.4/bin/compute-classpath.cmd
>
> Is there anyway to fix this?
>
> (Also, I have also tried running this from a normal terminal, instead of
> from cygwin, and I get the same issue, except this time the path is:
> C:\Users\jbeaudan\Spark\spark-1.3.1-bin-hadoop2.4\bin../bin/compute-classpath.cmd
> )
>
> Thank you!
>
> Julien
>
>
>
>
--
Best regards,
Elkhan Dadashov