You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Stuti Awasthi <st...@hcl.com> on 2014/08/19 07:05:47 UTC
Cannot run program "Rscript" using SparkR
Hi All,
I am using R 3.1 and Spark 0.9 and installed SparkR successfully. Now when I execute the "pi.R" example using spark master as local, then script executes fine.
But when I try to execute same example using master as spark cluster master, then in throws Rcript error.
Error :
java.io.IOException: Cannot run program "Rscript": java.io.IOException: error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
at edu.berkeley.cs.amplab.sparkr.RRDD.compute(RRDD.scala:113)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:109)
at org.apache.spark.scheduler.Task.run(Task.scala:53)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:213)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:46)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:45)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:45)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
I have checked, Rscript is present in my system and I also exported it in CLASSPATH and PATH variables. Script is given the permission 777 as there are multiple users of the clusters.
$ which Rscript
/usr/local/bin/Rscript
$ type -a Rscript
Rscript is /usr/local/bin/Rscript
$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/var/local/maven3/bin/:/var/local/ant/bin/:/usr/lib/jvm/java-6-openjdk:/usr/local/bin/Rscript
$ echo $CLASSPATH
:/usr/local/bin/Rscript
Also I am getting same error if I open R prompt and then execute the commands one after another OR if I execute the script.
Please suggest
Thanks
Stuti Awasthi
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
RE: Cannot run program "Rscript" using SparkR
Posted by Stuti Awasthi <st...@hcl.com>.
Thanks Shivaram,
This was the issue. Now I have installed Rscript on all the nodes in Spark cluster and it works now bith from script as well as R prompt.
Thanks
Stuti Awasthi
From: Shivaram Venkataraman [mailto:shivaram@eecs.berkeley.edu]
Sent: Tuesday, August 19, 2014 1:17 PM
To: Stuti Awasthi
Cc: user@spark.apache.org
Subject: Re: Cannot run program "Rscript" using SparkR
Hi Stuti
Could you check if Rscript is installed on all of the worker machines in the Spark cluster ? You can ssh into the machines and check if Rscript can be found in $PATH.
Thanks
Shivaram
On Mon, Aug 18, 2014 at 10:05 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi All,
I am using R 3.1 and Spark 0.9 and installed SparkR successfully. Now when I execute the “pi.R” example using spark master as local, then script executes fine.
But when I try to execute same example using master as spark cluster master, then in throws Rcript error.
Error :
java.io.IOException: Cannot run program "Rscript": java.io.IOException: error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
at edu.berkeley.cs.amplab.sparkr.RRDD.compute(RRDD.scala:113)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:109)
at org.apache.spark.scheduler.Task.run(Task.scala:53)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:213)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:46)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:45)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:45)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
I have checked, Rscript is present in my system and I also exported it in CLASSPATH and PATH variables. Script is given the permission 777 as there are multiple users of the clusters.
$ which Rscript
/usr/local/bin/Rscript
$ type -a Rscript
Rscript is /usr/local/bin/Rscript
$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/var/local/maven3/bin/:/var/local/ant/bin/:/usr/lib/jvm/java-6-openjdk:/usr/local/bin/Rscript
$ echo $CLASSPATH
:/usr/local/bin/Rscript
Also I am getting same error if I open R prompt and then execute the commands one after another OR if I execute the script.
Please suggest
Thanks
Stuti Awasthi
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
Re: Cannot run program "Rscript" using SparkR
Posted by Shivaram Venkataraman <sh...@eecs.berkeley.edu>.
Hi Stuti
Could you check if Rscript is installed on all of the worker machines in
the Spark cluster ? You can ssh into the machines and check if Rscript can
be found in $PATH.
Thanks
Shivaram
On Mon, Aug 18, 2014 at 10:05 PM, Stuti Awasthi <st...@hcl.com>
wrote:
> Hi All,
>
>
>
> I am using R 3.1 and Spark 0.9 and installed SparkR successfully. Now when
> I execute the “pi.R” example using spark master as local, then script
> executes fine.
>
> But when I try to execute same example using master as spark cluster
> master, then in throws Rcript error.
>
>
>
> Error :
>
> java.io.IOException: Cannot run program "Rscript": java.io.IOException:
> error=2, No such file or directory
>
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
>
> at edu.berkeley.cs.amplab.sparkr.RRDD.compute(RRDD.scala:113)
>
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
>
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
>
> at
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:109)
>
> at org.apache.spark.scheduler.Task.run(Task.scala:53)
>
> at
> org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:213)
>
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:46)
>
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:45)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:416)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>
> at
> org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:45)
>
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>
> at java.lang.Thread.run(Thread.java:679)
>
>
>
> I have checked, Rscript is present in my system and I also exported it in
> CLASSPATH and PATH variables. Script is given the permission 777 as there
> are multiple users of the clusters.
>
> $ which Rscript
>
> /usr/local/bin/Rscript
>
>
>
> $ type -a Rscript
>
> Rscript is /usr/local/bin/Rscript
>
>
>
> $ echo $PATH
>
>
> /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/var/local/maven3/bin/:/var/local/ant/bin/:/usr/lib/jvm/java-6-openjdk:/usr/local/bin/Rscript
>
>
>
> $ echo $CLASSPATH
>
> :/usr/local/bin/Rscript
>
>
>
> Also I am getting same error if I open R prompt and then execute the
> commands one after another OR if I execute the script.
>
>
>
> Please suggest
>
>
>
> Thanks
>
> Stuti Awasthi
>
>
>
>
>
>
>
> ::DISCLAIMER::
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> E-mail transmission is not guaranteed to be secure or error-free as
> information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or may contain viruses in
> transmission. The e mail and its contents
> (with or without referred errors) shall therefore not attach any liability
> on the originator or HCL or its affiliates.
> Views or opinions, if any, presented in this email are solely those of the
> author and may not necessarily reflect the
> views or opinions of HCL or its affiliates. Any form of reproduction,
> dissemination, copying, disclosure, modification,
> distribution and / or publication of this message without the prior
> written consent of authorized representative of
> HCL is strictly prohibited. If you have received this email in error
> please delete it and notify the sender immediately.
> Before opening any email and/or attachments, please check them for viruses
> and other defects.
>
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>