You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by luca_guerra <lg...@bitbang.com> on 2015/12/21 17:07:08 UTC
Problem with Spark Standalone
Hi,
I’m trying to submit a streaming application on my standalone Spark cluster,
this is my code:
import akka.actor.{Props, ActorSystem, Actor}
import akka.http.scaladsl.Http
import akka.http.scaladsl.model.HttpRequest
import akka.http.scaladsl.model.Uri
import akka.stream.ActorMaterializer
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.streaming.receiver.ActorHelper
import org.apache.spark.{SparkEnv, SparkContext, SparkConf}
object Boot extends App {
val driverPort = 7777
val conf = new SparkConf().
setAppName("TestSpark").
set("spark.driver.port", driverPort.toString).
set("spark.driver.host", "localhost").
setMaster("spark://<master-uri>:7077").
set("spark.akka.logLifecycleEvents", "true")
val sc = new SparkContext(conf)
val ssc = new StreamingContext(sc, Seconds(1))
val meetup = ssc.actorStream[String](Props[CustomReceiver], "attore")
ChunkReceiver.start(driverPort, "localhost", "attore")
meetup.print
ssc.start
}
object ChunkReceiver {
def start(driverPort:Int, driverHost:String, actorName:String) {
Thread.sleep(5000)
implicit val system = ActorSystem("client")
import system.dispatcher
implicit val materializer = ActorMaterializer()
val url =
s"akka.tcp://sparkDriver@$driverHost:$driverPort/user/Supervisor0/$actorName"
val actor = SparkEnv.get.actorSystem.actorSelection(url)
val source = Uri("http://stream.meetup.com/2/rsvps")
Http().
singleRequest(HttpRequest(uri = source)).
flatMap { response =>
response.entity.dataBytes.runForeach { chunk =>
actor ! chunk.utf8String
}
}
}
}
class CustomReceiver extends Actor with ActorHelper {
log.info("Starting receiver ...")
override def receive = {
case s:String => store(s)
}
}
And these are the messages generate by Spark:
15/12/21 06:43:24 WARN MetricsSystem: Using default name DAGScheduler for
source because spark.app.id is not set.
15/12/21 06:43:26 ERROR ErrorMonitor: AssociationError
[akka.tcp://sparkDriver@localhost:7777] <-
[akka.tcp://driverPropsFetcher@<ip-cluster>:51397]: Error [Shut down
address: akka.tcp://driverPropsFetcher@<ip-cluster>:51397] [
akka.remote.ShutDownAssociation: Shut down address:
akka.tcp://driverPropsFetcher@<ip-cluster>:51397
Caused by: akka.remote.transport.Transport$InvalidAssociationException: The
remote system terminated the association because it is shutting down.
]
akka.event.Logging$Error$NoCause$
15/12/21 06:43:26 ERROR EndpointWriter: AssociationError
[akka.tcp://sparkDriver@localhost:7777] <-
[akka.tcp://driverPropsFetcher@<ip-cluster>:51397]: Error [Shut down
address: akka.tcp://driverPropsFetcher@<ip-cluster>:51397] [
akka.remote.ShutDownAssociation: Shut down address:
akka.tcp://driverPropsFetcher@<ip-cluster>:51397
Caused by: akka.remote.transport.Transport$InvalidAssociationException: The
remote system terminated the association because it is shutting down.
]
15/12/21 06:43:26 ERROR ErrorMonitor: AssociationError
[akka.tcp://sparkDriver@localhost:7777] <-
[akka.tcp://driverPropsFetcher@<ip-cluster>:51463]: Error [Shut down
address: akka.tcp://driverPropsFetcher@<ip-cluster>:51463] [
akka.remote.ShutDownAssociation: Shut down address:
akka.tcp://driverPropsFetcher@<ip-cluster>:51463
Caused by: akka.remote.transport.Transport$InvalidAssociationException: The
remote system terminated the association because it is shutting down.
]
akka.event.Logging$Error$NoCause$
15/12/21 06:43:26 ERROR EndpointWriter: AssociationError
[akka.tcp://sparkDriver@localhost:7777] <-
[akka.tcp://driverPropsFetcher@<ip-cluster>:51463]: Error [Shut down
address: akka.tcp://driverPropsFetcher@<ip-cluster>:51463] [
akka.remote.ShutDownAssociation: Shut down address:
akka.tcp://driverPropsFetcher@<ip-cluster>:51463
Caused by: akka.remote.transport.Transport$InvalidAssociationException: The
remote system terminated the association because it is shutting down.
]
15/12/21 06:43:26 ERROR ErrorMonitor: AssociationError
[akka.tcp://sparkDriver@localhost:7777] <-
[akka.tcp://driverPropsFetcher@<ip-cluster>:59128]: Error [Shut down
address: akka.tcp://driverPropsFetcher@<ip-cluster>:59128] [
akka.remote.ShutDownAssociation: Shut down address:
akka.tcp://driverPropsFetcher@<ip-cluster>:59128
Caused by: akka.remote.transport.Transport$InvalidAssociationException: The
remote system terminated the association because it is shutting down.
]
akka.event.Logging$Error$NoCause$
15/12/21 06:43:26 ERROR EndpointWriter: AssociationError
[akka.tcp://sparkDriver@localhost:7777] <-
[akka.tcp://driverPropsFetcher@<ip-cluster>:59128]: Error [Shut down
address: akka.tcp://driverPropsFetcher@<ip-cluster>:59128] [
akka.remote.ShutDownAssociation: Shut down address:
akka.tcp://driverPropsFetcher@<ip-cluster>:59128
Caused by: akka.remote.transport.Transport$InvalidAssociationException: The
remote system terminated the association because it is shutting down.
]
the application doesn’t terminate with errors, but the CustomReceiver
doesn’t receive any data.
Thank you very much for the help,
Luca Guerra
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-with-Spark-Standalone-tp25750.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Problem with Spark Standalone
Posted by Vijay Gharge <vi...@gmail.com>.
Hi Luca
Are you able to run these commands from scala REPL shell ? Atleast 1 round
of iteration?
Looking at the error which says remote end shutting down - i suspect some
command in the code is triggering sc context shutdown or something similar.
One more point - if both master and executer both are on same server why
dont you paste complete code without any modification as well as spark gui
screenshot during the same time.
As asked in other mail - did this setup worked anytime in past ?
On Wednesday 23 December 2015, luca_guerra <lg...@bitbang.com> wrote:
> I don't think it's a "malformed IP address" issue because I have used an
> uri
> and not an IP.
> Another info, the master, driver and workers are hosted on the same machine
> so I use "localhost" as host for the Driver.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Problem-with-Spark-Standalone-tp25750p25787.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org <javascript:;>
> For additional commands, e-mail: user-help@spark.apache.org <javascript:;>
>
>
--
Regards,
Vijay Gharge
Re: Problem with Spark Standalone
Posted by luca_guerra <lg...@bitbang.com>.
I don't think it's a "malformed IP address" issue because I have used an uri
and not an IP.
Another info, the master, driver and workers are hosted on the same machine
so I use "localhost" as host for the Driver.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-with-Spark-Standalone-tp25750p25787.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Problem with Spark Standalone
Posted by MegaLearn <ji...@megalearningllc.com>.
I was going off this, not sure if it gives you a clue:
http://doc.akka.io/api/akka/2.4.0/index.html#akka.remote.transport.Transport$$InvalidAssociationException
"Indicates that the association setup request is invalid, and it is
impossible to recover (malformed IP address, hostname, etc.)."
I don't use akka so I don't know the ins and outs, I would ask the basic
debugging questions, did it work before? What changed? Sorry I can't be more
specific
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-with-Spark-Standalone-tp25750p25754.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Problem with Spark Standalone
Posted by luca_guerra <lg...@bitbang.com>.
This is the master's log file:
15/12/22 03:23:05 ERROR FileAppender: Error writing stream to file
/disco1/spark-1.5.1/work/app-20151222032252-0010/0/stderr
java.io.IOException: Stream closed
at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:272)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at
org.apache.spark.util.logging.FileAppender.appendStreamToFile(FileAppender.scala:70)
at
org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply$mcV$sp(FileAppender.scala:39)
at
org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
at
org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1699)
at
org.apache.spark.util.logging.FileAppender$$anon$1.run(FileAppender.scala:38)
15/12/22 03:23:05 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@<server-ip>:38095] has failed, address is
now gated for [5000] ms. Reason: [Disassociated]
And this is the worker's log file:
15/12/22 03:23:05 ERROR FileAppender: Error writing stream to file
/disco1/spark-1.5.1/work/app-20151222032252-0010/2/stderr
java.io.IOException: Stream closed
at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:272)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at
org.apache.spark.util.logging.FileAppender.appendStreamToFile(FileAppender.scala:70)
at
org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply$mcV$sp(FileAppender.scala:39)
at
org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
at
org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1699)
at
org.apache.spark.util.logging.FileAppender$$anon$1.run(FileAppender.scala:38)
15/12/22 03:23:05 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@<server-ip>:41370] has failed, address is
now gated for [5000] ms. Reason: [Disassociated]
Thank you very much for the help.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-with-Spark-Standalone-tp25750p25786.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Problem with Spark Standalone
Posted by MegaLearn <ji...@megalearningllc.com>.
Gotcha, then you are also replacing the cluster IP. Missed that.
I would ask you to post the actual logfiles, not sure I'll be able to help
but hopefully it gives more info that someone can work with :)
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-with-Spark-Standalone-tp25750p25753.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Problem with Spark Standalone
Posted by luca_guerra <lg...@bitbang.com>.
Hi MegaLearn!
thanks for the reply!
<master-uri> it's a placeholder, in my real application I use the right
master's hostname.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-with-Spark-Standalone-tp25750p25752.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Problem with Spark Standalone
Posted by MegaLearn <ji...@megalearningllc.com>.
Perhaps put the master IP address in this line and try again?
setMaster("spark://<master-uri>:7077").
Replace <master-uri> with hostname, but the way our host files are setup I
have to put the IP address there.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-with-Spark-Standalone-tp25750p25751.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org