SparkR failed to connect to the master

[cjwang <cj...@cjwang.us>]

I have a cluster running.  I was able to run Spark Shell and submit programs. 
But when I tried to use SparkR, I got these errors:

wifi-orcus:sparkR cwang$ MASTER=spark://wifi-orcus.dhcp.carrieriq.com:7077
sparkR

R version 3.1.0 (2014-04-10) -- "Spring Dance"
Copyright (C) 2014 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin13.1.0 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

[Previously saved workspace restored]

Loading required package: SparkR
Loading required package: rJava
[SparkR] Initializing with classpath
/Users/cwang/amplab-extras-SparkR-pkg-e1f95b6/lib/SparkR/sparkr-assembly-0.1.jar

14/07/10 15:54:31 INFO Slf4jLogger: Slf4jLogger started
2014-07-10 15:54:32.391 R[19021:d07] Unable to load realm info from
SCDynamicStore

 Welcome to SparkR!
 Spark context is available as sc
> 14/07/10 15:55:32 ERROR AppClient$ClientActor: All masters are
> unresponsive! Giving up.
14/07/10 15:55:32 ERROR SparkDeploySchedulerBackend: Spark cluster looks
dead, giving up.
14/07/10 15:55:32 ERROR TaskSchedulerImpl: Exiting due to error from cluster
scheduler: Spark cluster looks down
Error: Java called System.exit(1) requesting R to quit - trying to recover
> 


Then it crashed.  The webui of the master does not show any application
connections from SaprkR.  The cluster was not dead.  I was able to launch
Spark Shell and do something after that.

What was wrong here?





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkR-failed-to-connect-to-the-master-tp9359.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: SparkR failed to connect to the master

[Shivaram Venkataraman <sh...@eecs.berkeley.edu>]

You'll need to build SparkR to match the Spark version deployed on the
cluster. You can do that by changing the Spark version in SparkR's
build.sbt [1]. If you are using the Maven build you'll need to edit pom.xml

Thanks
Shivaram

[1]
https://github.com/amplab-extras/SparkR-pkg/blob/master/pkg/src/build.sbt#L20


On Mon, Jul 14, 2014 at 6:19 PM, cjwang <cj...@cjwang.us> wrote:

> I tried installing the latest Spark 1.0.1 and SparkR couldn't find the
> master
> either.  I restarted with Spark 0.9.1 and SparkR was able to find the
> master.  So, there seemed to be something that changed after Spark 1.0.0.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/SparkR-failed-to-connect-to-the-master-tp9359p9680.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>

Re: SparkR failed to connect to the master

[cjwang <cj...@cjwang.us>]

I tried installing the latest Spark 1.0.1 and SparkR couldn't find the master
either.  I restarted with Spark 0.9.1 and SparkR was able to find the
master.  So, there seemed to be something that changed after Spark 1.0.0.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkR-failed-to-connect-to-the-master-tp9359p9680.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: SparkR failed to connect to the master

[cjwang <cj...@cjwang.us>]

I restarted Spark Master with spark-0.9.1 and SparkR was able to communicate
with the Master.  I am using the latest SparkR pkg-e1f95b6.  Maybe it has
problem communicating to Spark 1.0.0?





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkR-failed-to-connect-to-the-master-tp9359p9658.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.