You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Grega Kešpret <gr...@celtra.com> on 2013/11/25 11:53:38 UTC
spark-shell not working on standalone cluster (java.io.IOException:
Cannot run program compute-classpath.sh)
Hi,
I'm trying to run spark-shell and point it to Spark standalone cluster. I
have Spark locally on a different directory than on cluster. Locally, I
have it in "/home/grega/mab/analyzer/target/spark" and on the cluster I
have it in "/opt/spark".
When I run the spark-shell script with:
SPARK_HOME="/opt/spark" SPARK_PRINT_LAUNCH_COMMAND=1
SPARK_HADOOP_VERSION=1.1.0 SPARK_MEM="5g" MASTER="spark://..." spark-shell
it fails with
13/11/25 10:46:24 INFO server.Server: jetty-7.x.y-SNAPSHOT
13/11/25 10:46:24 INFO server.AbstractConnector: Started
SocketConnector@0.0.0.0:43729
13/11/25 10:46:24 INFO broadcast.HttpBroadcast: Broadcast server started at
http://10.170.8.11:43729
13/11/25 10:46:24 INFO spark.SparkEnv: Registering MapOutputTracker
13/11/25 10:46:24 INFO spark.HttpFileServer: HTTP File server directory is
/tmp/spark-aa291810-da5e-4d09-8cef-d025975813d7
13/11/25 10:46:24 INFO server.Server: jetty-7.x.y-SNAPSHOT
13/11/25 10:46:24 INFO server.AbstractConnector: Started
SocketConnector@0.0.0.0:44042
13/11/25 10:46:24 INFO server.Server: jetty-7.x.y-SNAPSHOT
13/11/25 10:46:24 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/storage/rdd,null}
13/11/25 10:46:24 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/storage,null}
13/11/25 10:46:24 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/stages/stage,null}
13/11/25 10:46:24 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/stages/pool,null}
13/11/25 10:46:24 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/stages,null}
13/11/25 10:46:24 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/environment,null}
13/11/25 10:46:24 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/executors,null}
13/11/25 10:46:24 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/metrics/json,null}
13/11/25 10:46:24 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/static,null}
13/11/25 10:46:24 INFO handler.ContextHandler: started
o.e.j.s.h.ContextHandler{/,null}
13/11/25 10:46:24 INFO server.AbstractConnector: Started
SelectChannelConnector@0.0.0.0:4040
13/11/25 10:46:24 INFO ui.SparkUI: Started Spark Web UI at
http://ip-10-170-8-11.ec2.internal:4040
13/11/25 10:46:24 INFO client.Client$ClientActor: Connecting to master
spark://ec2-107-20-48-235.compute-1.amazonaws.com:7077
Spark context available as sc.
13/11/25 10:46:24 INFO cluster.SparkDeploySchedulerBackend: Connected to
Spark cluster with app ID app-20131125104624-0028
13/11/25 10:46:24 INFO client.Client$ClientActor: Executor added:
app-20131125104624-0028/0 on
worker-20131125094226-ip-10-235-17-154.ec2.internal-49711
(ip-10-235-17-154.ec2.internal:49711) with 8 cores
13/11/25 10:46:24 INFO cluster.SparkDeploySchedulerBackend: Granted
executor ID app-20131125104624-0028/0 on hostPort
ip-10-235-17-154.ec2.internal:49711 with 8 cores, 5.0 GB RAM
13/11/25 10:46:25 INFO client.Client$ClientActor: Executor updated:
app-20131125104624-0028/0 is now RUNNING
*13/11/25 10:46:25 INFO client.Client$ClientActor: Executor updated:
app-20131125104624-0028/0 is now FAILED (class java.io.IOException: Cannot
run program
"/home/grega/mab/analyzer/target/spark/bin/compute-classpath.sh" (in
directory "."): java.io.IOException: error=2, No such file or directory)*
13/11/25 10:46:25 INFO cluster.SparkDeploySchedulerBackend: Executor
app-20131125104624-0028/0 removed: class java.io.IOException: Cannot run
program "/home/grega/mab/analyzer/target/spark/bin/compute-classpath.sh"
(in directory "."): java.io.IOException: error=2, No such file or directory
....
13/11/25 10:46:25 ERROR client.Client$ClientActor: Master removed our
application: FAILED; stopping client
13/11/25 10:46:25 ERROR cluster.SparkDeploySchedulerBackend: Disconnected
from Spark cluster!
13/11/25 10:46:25 ERROR cluster.ClusterScheduler: Exiting due to error from
cluster scheduler: Disconnected from Spark cluster
It seems to me that it tries to run compute-classpath.sh on a worker node
from wrong Spark path, even though I am passing SPARK_HOME through
environment variable on the driver node. Any help appreciated.
Thanks,
Grega
--
[image: Inline image 1]
*Grega Kešpret*
Analytics engineer
Celtra — Rich Media Mobile Advertising
celtra.com <http://www.celtra.com/> |
@celtramobile<http://www.twitter.com/celtramobile>
Re: spark-shell not working on standalone cluster (java.io.IOException:
Cannot run program compute-classpath.sh)
Posted by Grega Kešpret <gr...@celtra.com>.
Thanks, will try it out!
Grega
--
[image: Inline image 1]
*Grega Kešpret*
Analytics engineer
Celtra — Rich Media Mobile Advertising
celtra.com <http://www.celtra.com/> |
@celtramobile<http://www.twitter.com/celtramobile>
On Mon, Nov 25, 2013 at 11:54 PM, Aaron Davidson <il...@gmail.com> wrote:
> There is a pull request currently to fix this exact issue, I believe, at
> https://github.com/apache/incubator-spark/pull/192. It's very small and
> only touches the script files, so you could apply it to your current
> version and distribute it to the workers. The fix here is that you add an
> additional variable in spark-env that specifies the REMOTE_SPARK_HOME.
>
>
> On Mon, Nov 25, 2013 at 3:40 AM, Grega Kešpret <gr...@celtra.com> wrote:
>
>> It seems there is already an open ticket for this -
>> https://spark-project.atlassian.net/browse/SPARK-905 , but for version
>> 0.7.3.
>>
>> Grega
>> --
>> [image: Inline image 1]
>> *Grega Kešpret*
>> Analytics engineer
>>
>> Celtra — Rich Media Mobile Advertising
>> celtra.com <http://www.celtra.com/> | @celtramobile<http://www.twitter.com/celtramobile>
>>
>>
>> On Mon, Nov 25, 2013 at 12:13 PM, Grega Kešpret <gr...@celtra.com> wrote:
>>
>>> Sorry, forgot to mention, I run spark version "v0.8.0-incubating" from
>>> https://github.com/apache/incubator-spark.git.
>>> It seems to work when local Spark directory is also /opt/spark, so I
>>> think this confirms my doubt that SPARK_HOME somehow doesn't get passed to
>>> the Executor?
>>>
>>> Grega
>>> --
>>> [image: Inline image 1]
>>> *Grega Kešpret*
>>> Analytics engineer
>>>
>>> Celtra — Rich Media Mobile Advertising
>>> celtra.com <http://www.celtra.com/> | @celtramobile<http://www.twitter.com/celtramobile>
>>>
>>
>>
>
Re: spark-shell not working on standalone cluster (java.io.IOException:
Cannot run program compute-classpath.sh)
Posted by Aaron Davidson <il...@gmail.com>.
There is a pull request currently to fix this exact issue, I believe, at
https://github.com/apache/incubator-spark/pull/192. It's very small and
only touches the script files, so you could apply it to your current
version and distribute it to the workers. The fix here is that you add an
additional variable in spark-env that specifies the REMOTE_SPARK_HOME.
On Mon, Nov 25, 2013 at 3:40 AM, Grega Kešpret <gr...@celtra.com> wrote:
> It seems there is already an open ticket for this -
> https://spark-project.atlassian.net/browse/SPARK-905 , but for version
> 0.7.3.
>
> Grega
> --
> [image: Inline image 1]
> *Grega Kešpret*
> Analytics engineer
>
> Celtra — Rich Media Mobile Advertising
> celtra.com <http://www.celtra.com/> | @celtramobile<http://www.twitter.com/celtramobile>
>
>
> On Mon, Nov 25, 2013 at 12:13 PM, Grega Kešpret <gr...@celtra.com> wrote:
>
>> Sorry, forgot to mention, I run spark version "v0.8.0-incubating" from
>> https://github.com/apache/incubator-spark.git.
>> It seems to work when local Spark directory is also /opt/spark, so I
>> think this confirms my doubt that SPARK_HOME somehow doesn't get passed to
>> the Executor?
>>
>> Grega
>> --
>> [image: Inline image 1]
>> *Grega Kešpret*
>> Analytics engineer
>>
>> Celtra — Rich Media Mobile Advertising
>> celtra.com <http://www.celtra.com/> | @celtramobile<http://www.twitter.com/celtramobile>
>>
>
>
Re: spark-shell not working on standalone cluster (java.io.IOException:
Cannot run program compute-classpath.sh)
Posted by Grega Kešpret <gr...@celtra.com>.
It seems there is already an open ticket for this -
https://spark-project.atlassian.net/browse/SPARK-905 , but for version
0.7.3.
Grega
--
[image: Inline image 1]
*Grega Kešpret*
Analytics engineer
Celtra — Rich Media Mobile Advertising
celtra.com <http://www.celtra.com/> |
@celtramobile<http://www.twitter.com/celtramobile>
On Mon, Nov 25, 2013 at 12:13 PM, Grega Kešpret <gr...@celtra.com> wrote:
> Sorry, forgot to mention, I run spark version "v0.8.0-incubating" from
> https://github.com/apache/incubator-spark.git.
> It seems to work when local Spark directory is also /opt/spark, so I think
> this confirms my doubt that SPARK_HOME somehow doesn't get passed to the
> Executor?
>
> Grega
> --
> [image: Inline image 1]
> *Grega Kešpret*
> Analytics engineer
>
> Celtra — Rich Media Mobile Advertising
> celtra.com <http://www.celtra.com/> | @celtramobile<http://www.twitter.com/celtramobile>
>
Re: spark-shell not working on standalone cluster (java.io.IOException:
Cannot run program compute-classpath.sh)
Posted by Grega Kešpret <gr...@celtra.com>.
Sorry, forgot to mention, I run spark version "v0.8.0-incubating" from
https://github.com/apache/incubator-spark.git.
It seems to work when local Spark directory is also /opt/spark, so I think
this confirms my doubt that SPARK_HOME somehow doesn't get passed to the
Executor?
Grega
--
[image: Inline image 1]
*Grega Kešpret*
Analytics engineer
Celtra — Rich Media Mobile Advertising
celtra.com <http://www.celtra.com/> |
@celtramobile<http://www.twitter.com/celtramobile>