You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jai <ja...@gmail.com> on 2014/12/16 08:55:03 UTC
Accessing Apache Spark from Java
Hi
I have installed a standalone Spark set up in standalone mode in a Linux
server and I am trying to access that spark setup from Java in windows. When
I try connecting to Spark I see the following exception
14/12/16 12:52:52 WARN TaskSchedulerImpl: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and
have sufficient memory
14/12/16 12:52:56 INFO AppClient$ClientActor: Connecting to master
spark://01hw294954.INDIA:7077...
14/12/16 12:53:07 WARN TaskSchedulerImpl: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and
have sufficient memory
14/12/16 12:53:16 INFO AppClient$ClientActor: Connecting to master
spark://01hw294954.INDIA:7077...
14/12/16 12:53:22 WARN TaskSchedulerImpl: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and
have sufficient memory
14/12/16 12:53:36 ERROR SparkDeploySchedulerBackend: Application has been
killed. Reason: All masters are unresponsive! Giving up.
14/12/16 12:53:36 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks
have all completed, from pool
14/12/16 12:53:36 INFO TaskSchedulerImpl: Cancelling stage 0
14/12/16 12:53:36 INFO DAGScheduler: Failed to run collect at
MySqlConnector.java:579
Exception in thread "main" org.apache.spark.SparkException: Job aborted due
to stage failure: All masters are unresponsive! Giving up.
at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1033)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1017)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1015)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1015)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:633)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:633)
at scala.Option.foreach(Option.scala:236)
at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:633)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1207)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run
I have attached the Spark Master UI
Spark Master at spark://01hw294954.INDIA:7077
URL: spark://01hw294954.INDIA:7077
Workers: 1
Cores: 2 Total, 0 Used
Memory: 835.0 MB Total, 0.0 B Used
Applications: 0 Running, 0 Completed
Drivers: 0 Running, 0 Completed
Status: ALIVE
Workers
Id Address State Cores Memory
worker-20141216123503-01hw294954.INDIA-38962 01hw294954.INDIA:38962 ALIVE 2
(0 Used) 835.0 MB (0.0 B Used)
Running Applications
ID Name Cores Memory per Node Submitted Time User State Duration
Completed Applications
ID Name Cores Memory per Node Submitted Time User State Duration
My Spark Slave is
Spark Worker at 01hw294954.INDIA:38962
ID: worker-20141216123503-01hw294954.INDIA-38962
Master URL: spark://01hw294954.INDIA:7077
Cores: 2 (0 Used)
Memory: 835.0 MB (0.0 B Used)
Back to Master
Running Executors (0)
ExecutorID Cores State Memory Job Details Logs
My Java Master Code looks like this
SparkConf sparkConf = new SparkConf().setAppName("JdbcRddTest");
sparkConf.setMaster("spark://01hw294954.INDIA:7077");
When I tried using the same code with the local spark set up as the master
it ran.
Any help for solving this issue is very much appreciated.
Thanks and Regards
Jai
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Accessing-Apache-Spark-from-Java-tp20700.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Accessing Apache Spark from Java
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Hi Jai,
Refer this doc and make sure your network is not blocking
http://apache-spark-user-list.1001560.n3.nabble.com/Submitting-Spark-job-on-Unix-cluster-from-dev-environment-Windows-td16989.html
Also make sure you are using the same version of spark in both places (the
one on the cluster, and the one that you used inside your application)
Thanks
Best Regards
On Tue, Dec 16, 2014 at 1:25 PM, Jai <ja...@gmail.com> wrote:
>
> Hi
>
> I have installed a standalone Spark set up in standalone mode in a Linux
> server and I am trying to access that spark setup from Java in windows.
> When
> I try connecting to Spark I see the following exception
>
> 14/12/16 12:52:52 WARN TaskSchedulerImpl: Initial job has not accepted any
> resources; check your cluster UI to ensure that workers are registered and
> have sufficient memory
> 14/12/16 12:52:56 INFO AppClient$ClientActor: Connecting to master
> spark://01hw294954.INDIA:7077...
> 14/12/16 12:53:07 WARN TaskSchedulerImpl: Initial job has not accepted any
> resources; check your cluster UI to ensure that workers are registered and
> have sufficient memory
> 14/12/16 12:53:16 INFO AppClient$ClientActor: Connecting to master
> spark://01hw294954.INDIA:7077...
> 14/12/16 12:53:22 WARN TaskSchedulerImpl: Initial job has not accepted any
> resources; check your cluster UI to ensure that workers are registered and
> have sufficient memory
> 14/12/16 12:53:36 ERROR SparkDeploySchedulerBackend: Application has been
> killed. Reason: All masters are unresponsive! Giving up.
> 14/12/16 12:53:36 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks
> have all completed, from pool
> 14/12/16 12:53:36 INFO TaskSchedulerImpl: Cancelling stage 0
> 14/12/16 12:53:36 INFO DAGScheduler: Failed to run collect at
> MySqlConnector.java:579
> Exception in thread "main" org.apache.spark.SparkException: Job aborted due
> to stage failure: All masters are unresponsive! Giving up.
> at
> org.apache.spark.scheduler.DAGScheduler.org
> $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1033)
> at
>
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1017)
> at
>
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1015)
> at
>
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at
> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> at
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1015)
> at
>
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:633)
> at
>
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:633)
> at scala.Option.foreach(Option.scala:236)
> at
>
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:633)
> at
>
> org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1207)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
> at akka.actor.ActorCell.invoke(ActorCell.scala:456)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
> at akka.dispatch.Mailbox.run(Mailbox.scala:219)
> at
>
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
> at
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
>
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run
>
> I have attached the Spark Master UI
>
> Spark Master at spark://01hw294954.INDIA:7077
> URL: spark://01hw294954.INDIA:7077
> Workers: 1
> Cores: 2 Total, 0 Used
> Memory: 835.0 MB Total, 0.0 B Used
> Applications: 0 Running, 0 Completed
> Drivers: 0 Running, 0 Completed
> Status: ALIVE
> Workers
>
> Id Address State Cores Memory
> worker-20141216123503-01hw294954.INDIA-38962 01hw294954.INDIA:38962
> ALIVE 2
> (0 Used) 835.0 MB (0.0 B Used)
> Running Applications
>
> ID Name Cores Memory per Node Submitted Time User State
> Duration
> Completed Applications
>
> ID Name Cores Memory per Node Submitted Time User State
> Duration
>
>
> My Spark Slave is
>
> Spark Worker at 01hw294954.INDIA:38962
> ID: worker-20141216123503-01hw294954.INDIA-38962
> Master URL: spark://01hw294954.INDIA:7077
> Cores: 2 (0 Used)
> Memory: 835.0 MB (0.0 B Used)
> Back to Master
>
> Running Executors (0)
>
> ExecutorID Cores State Memory Job Details Logs
>
>
> My Java Master Code looks like this
>
> SparkConf sparkConf = new SparkConf().setAppName("JdbcRddTest");
> sparkConf.setMaster("spark://01hw294954.INDIA:7077");
> When I tried using the same code with the local spark set up as the master
> it ran.
>
> Any help for solving this issue is very much appreciated.
>
> Thanks and Regards
> Jai
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Accessing-Apache-Spark-from-Java-tp20700.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>