You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Krishnanand Khambadkone <kk...@yahoo.com.INVALID> on 2015/03/02 23:06:43 UTC
Spark Error: Cause was: akka.remote.InvalidAssociation: Invalid
address: akka.tcp://sparkMaster@localhost:7077
Hi, I am running spark on my mac. It is reading from a kafka topic and then writes the data to a hbase table. When I do a spark submit, I get this error,
Error connecting to master spark://localhost:7077 (akka.tcp://sparkMaster@localhost:7077), exiting.
Cause was: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@localhost:7077
My submit statement looks like this,
./spark-submit --jars /Users/hadoop/kafka-0.8.1.1-src/core/build/libs/kafka_2.8.0-0.8.1.1.jar --class KafkaMain --master spark://localhost:7077 --deploy-mode cluster --num-executors 100 --driver-memory 1g --executor-memory 1g /Users/hadoop/dev/kafkahbase/kafka/src/sparkhbase.jar
Re: Spark Error: Cause was: akka.remote.InvalidAssociation: Invalid
address: akka.tcp://sparkMaster@localhost:7077
Posted by Krishnanand Khambadkone <kk...@yahoo.com.INVALID>.
Hello Ted, Some progress, now it seems that the spark job does get submitted, In the spark web UI, I do see this under finished drivers. However, it seems to not go past this step, JavaPairReceiverInputDStream<String, String> messages = KafkaUtils.createStream(jsc, "localhost:2181", "aa", topicMap); I do have several messages in the Kafka topic. When I run the consumer with from-beginning option, I do see all the messages.
krishs-mbp:bin hadoop$ ./spark-submit --jars /Users/hadoop/kafka-0.8.1.1-src/core/build/libs/kafka_2.8.0-0.8.1.1.jar --class KafkaMain --master spark://krishs-mbp:7077 --deploy-mode cluster --num-executors 100 --driver-memory 1g --executor-memory 1g --supervise file:///Users/hadoop/dev/kafkahbase/kafka/src/sparkhbase.jar
Sending launch command to spark://krishs-mbp:7077
Driver successfully submitted as driver-20150303085115-0007
... waiting before polling master for driver state
... polling master for driver state
State of driver-20150303085115-0007 is FINISHED
On Monday, March 2, 2015 4:31 PM, Ted Yu <yu...@gmail.com> wrote:
In AkkaUtils.scala:
val akkaLogLifecycleEvents = conf.getBoolean("spark.akka.logLifecycleEvents", false)
Can you turn on life cycle event logging to see if you would get some more clue ?
Cheers
On Mon, Mar 2, 2015 at 3:56 PM, Krishnanand Khambadkone <kk...@yahoo.com> wrote:
I see these messages now,
spark.master -> spark://krishs-mbp:7077
Classpath elements:
Sending launch command to spark://krishs-mbp:7077
Driver successfully submitted as driver-20150302155433-0000
... waiting before polling master for driver state
... polling master for driver state
State of driver-20150302155433-0000 is FAILED
On Monday, March 2, 2015 2:42 PM, Ted Yu <yu...@gmail.com> wrote:
bq. Cause was: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@localhost:7077
There should be some more output following the above line.
Can you post them ?
Cheers
On Mon, Mar 2, 2015 at 2:06 PM, Krishnanand Khambadkone <kk...@yahoo.com.invalid> wrote:
Hi, I am running spark on my mac. It is reading from a kafka topic and then writes the data to a hbase table. When I do a spark submit, I get this error,
Error connecting to master spark://localhost:7077 (akka.tcp://sparkMaster@localhost:7077), exiting.
Cause was: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@localhost:7077
My submit statement looks like this,
./spark-submit --jars /Users/hadoop/kafka-0.8.1.1-src/core/build/libs/kafka_2.8.0-0.8.1.1.jar --class KafkaMain --master spark://localhost:7077 --deploy-mode cluster --num-executors 100 --driver-memory 1g --executor-memory 1g /Users/hadoop/dev/kafkahbase/kafka/src/sparkhbase.jar
Re: Spark Error: Cause was: akka.remote.InvalidAssociation: Invalid
address: akka.tcp://sparkMaster@localhost:7077
Posted by Ted Yu <yu...@gmail.com>.
In AkkaUtils.scala:
val akkaLogLifecycleEvents =
conf.getBoolean("spark.akka.logLifecycleEvents", false)
Can you turn on life cycle event logging to see if you would get some more
clue ?
Cheers
On Mon, Mar 2, 2015 at 3:56 PM, Krishnanand Khambadkone <
kkhambadkone@yahoo.com> wrote:
> I see these messages now,
>
> spark.master -> spark://krishs-mbp:7077
> Classpath elements:
>
>
>
> Sending launch command to spark://krishs-mbp:7077
> Driver successfully submitted as driver-20150302155433-0000
> ... waiting before polling master for driver state
> ... polling master for driver state
> State of driver-20150302155433-0000 is FAILED
>
>
> On Monday, March 2, 2015 2:42 PM, Ted Yu <yu...@gmail.com> wrote:
>
>
> bq. Cause was: akka.remote.InvalidAssociation: Invalid address:
> akka.tcp://sparkMaster@localhost:7077
>
> There should be some more output following the above line.
>
> Can you post them ?
>
> Cheers
>
> On Mon, Mar 2, 2015 at 2:06 PM, Krishnanand Khambadkone <
> kkhambadkone@yahoo.com.invalid> wrote:
>
> Hi, I am running spark on my mac. It is reading from a kafka topic and
> then writes the data to a hbase table. When I do a spark submit, I get
> this error,
>
> Error connecting to master spark://localhost:7077
> (akka.tcp://sparkMaster@localhost:7077), exiting.
> Cause was: akka.remote.InvalidAssociation: Invalid address:
> akka.tcp://sparkMaster@localhost:7077
>
> My submit statement looks like this,
>
> ./spark-submit --jars
> /Users/hadoop/kafka-0.8.1.1-src/core/build/libs/kafka_2.8.0-0.8.1.1.jar
> --class KafkaMain --master spark://localhost:7077 --deploy-mode cluster
> --num-executors 100 --driver-memory 1g --executor-memory 1g
> /Users/hadoop/dev/kafkahbase/kafka/src/sparkhbase.jar
>
>
>
>
>
Re: Spark Error: Cause was: akka.remote.InvalidAssociation: Invalid
address: akka.tcp://sparkMaster@localhost:7077
Posted by Krishnanand Khambadkone <kk...@yahoo.com.INVALID>.
I see these messages now,
spark.master -> spark://krishs-mbp:7077
Classpath elements:
Sending launch command to spark://krishs-mbp:7077
Driver successfully submitted as driver-20150302155433-0000
... waiting before polling master for driver state
... polling master for driver state
State of driver-20150302155433-0000 is FAILED
On Monday, March 2, 2015 2:42 PM, Ted Yu <yu...@gmail.com> wrote:
bq. Cause was: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@localhost:7077
There should be some more output following the above line.
Can you post them ?
Cheers
On Mon, Mar 2, 2015 at 2:06 PM, Krishnanand Khambadkone <kk...@yahoo.com.invalid> wrote:
Hi, I am running spark on my mac. It is reading from a kafka topic and then writes the data to a hbase table. When I do a spark submit, I get this error,
Error connecting to master spark://localhost:7077 (akka.tcp://sparkMaster@localhost:7077), exiting.
Cause was: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@localhost:7077
My submit statement looks like this,
./spark-submit --jars /Users/hadoop/kafka-0.8.1.1-src/core/build/libs/kafka_2.8.0-0.8.1.1.jar --class KafkaMain --master spark://localhost:7077 --deploy-mode cluster --num-executors 100 --driver-memory 1g --executor-memory 1g /Users/hadoop/dev/kafkahbase/kafka/src/sparkhbase.jar
Re: Spark Error: Cause was: akka.remote.InvalidAssociation: Invalid
address: akka.tcp://sparkMaster@localhost:7077
Posted by Krishnanand Khambadkone <kk...@yahoo.com.INVALID>.
I ran it with the --verbose option and I see this output
Using properties file: null
Parsed arguments:
master spark://localhost:7077
deployMode cluster
executorMemory 1g
executorCores null
totalExecutorCores null
propertiesFile null
driverMemory 1g
driverCores null
driverExtraClassPath null
driverExtraLibraryPath null
driverExtraJavaOptions null
supervise false
queue null
numExecutors 100
files null
pyFiles null
archives null
mainClass KafkaMain
primaryResource file:/Users/hadoop/dev/kafkahbase/kafka/src/sparkhbase.jar
name KafkaMain
childArgs []
jars file:/Users/hadoop/kafka-0.8.1.1-src/core/build/libs/kafka_2.8.0-0.8.1.1.jar
verbose true
Spark properties used, including those specified through
--conf and those from the properties file null:
Main class:
org.apache.spark.deploy.Client
Arguments:
--memory
1g
launch
spark://localhost:7077
file:/Users/hadoop/dev/kafkahbase/kafka/src/sparkhbase.jar
KafkaMain
System properties:
spark.executor.memory -> 1g
SPARK_SUBMIT -> true
spark.app.name -> KafkaMain
spark.jars -> file:/Users/hadoop/kafka-0.8.1.1-src/core/build/libs/kafka_2.8.0-0.8.1.1.jar,file:/Users/hadoop/dev/kafkahbase/kafka/src/sparkhbase.jar
spark.master -> spark://localhost:7077
Classpath elements:
On Monday, March 2, 2015 2:42 PM, Ted Yu <yu...@gmail.com> wrote:
bq. Cause was: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@localhost:7077
There should be some more output following the above line.
Can you post them ?
Cheers
On Mon, Mar 2, 2015 at 2:06 PM, Krishnanand Khambadkone <kk...@yahoo.com.invalid> wrote:
Hi, I am running spark on my mac. It is reading from a kafka topic and then writes the data to a hbase table. When I do a spark submit, I get this error,
Error connecting to master spark://localhost:7077 (akka.tcp://sparkMaster@localhost:7077), exiting.
Cause was: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@localhost:7077
My submit statement looks like this,
./spark-submit --jars /Users/hadoop/kafka-0.8.1.1-src/core/build/libs/kafka_2.8.0-0.8.1.1.jar --class KafkaMain --master spark://localhost:7077 --deploy-mode cluster --num-executors 100 --driver-memory 1g --executor-memory 1g /Users/hadoop/dev/kafkahbase/kafka/src/sparkhbase.jar
Re: Spark Error: Cause was: akka.remote.InvalidAssociation: Invalid
address: akka.tcp://sparkMaster@localhost:7077
Posted by Krishnanand Khambadkone <kk...@yahoo.com.INVALID>.
This is the line,
Error connecting to master spark://localhost:7077 (akka.tcp://sparkMaster@localhost:7077), exiting.
On Monday, March 2, 2015 2:42 PM, Ted Yu <yu...@gmail.com> wrote:
bq. Cause was: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@localhost:7077
There should be some more output following the above line.
Can you post them ?
Cheers
On Mon, Mar 2, 2015 at 2:06 PM, Krishnanand Khambadkone <kk...@yahoo.com.invalid> wrote:
Hi, I am running spark on my mac. It is reading from a kafka topic and then writes the data to a hbase table. When I do a spark submit, I get this error,
Error connecting to master spark://localhost:7077 (akka.tcp://sparkMaster@localhost:7077), exiting.
Cause was: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@localhost:7077
My submit statement looks like this,
./spark-submit --jars /Users/hadoop/kafka-0.8.1.1-src/core/build/libs/kafka_2.8.0-0.8.1.1.jar --class KafkaMain --master spark://localhost:7077 --deploy-mode cluster --num-executors 100 --driver-memory 1g --executor-memory 1g /Users/hadoop/dev/kafkahbase/kafka/src/sparkhbase.jar
Re: Spark Error: Cause was: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@localhost:7077
Posted by Krish Khambadkone <kk...@yahoo.com.INVALID>.
There is no output after this line
Sent from my iPhone
> On Mar 2, 2015, at 2:40 PM, Ted Yu <yu...@gmail.com> wrote:
>
> bq. Cause was: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@localhost:7077
>
> There should be some more output following the above line.
>
> Can you post them ?
>
> Cheers
>
>> On Mon, Mar 2, 2015 at 2:06 PM, Krishnanand Khambadkone <kk...@yahoo.com.invalid> wrote:
>> Hi, I am running spark on my mac. It is reading from a kafka topic and then writes the data to a hbase table. When I do a spark submit, I get this error,
>>
>> Error connecting to master spark://localhost:7077 (akka.tcp://sparkMaster@localhost:7077), exiting.
>> Cause was: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@localhost:7077
>>
>> My submit statement looks like this,
>>
>> ./spark-submit --jars /Users/hadoop/kafka-0.8.1.1-src/core/build/libs/kafka_2.8.0-0.8.1.1.jar --class KafkaMain --master spark://localhost:7077 --deploy-mode cluster --num-executors 100 --driver-memory 1g --executor-memory 1g /Users/hadoop/dev/kafkahbase/kafka/src/sparkhbase.jar
>
Re: Spark Error: Cause was: akka.remote.InvalidAssociation: Invalid
address: akka.tcp://sparkMaster@localhost:7077
Posted by Ted Yu <yu...@gmail.com>.
bq. Cause was: akka.remote.InvalidAssociation: Invalid address:
akka.tcp://sparkMaster@localhost:7077
There should be some more output following the above line.
Can you post them ?
Cheers
On Mon, Mar 2, 2015 at 2:06 PM, Krishnanand Khambadkone <
kkhambadkone@yahoo.com.invalid> wrote:
> Hi, I am running spark on my mac. It is reading from a kafka topic and
> then writes the data to a hbase table. When I do a spark submit, I get
> this error,
>
> Error connecting to master spark://localhost:7077
> (akka.tcp://sparkMaster@localhost:7077), exiting.
> Cause was: akka.remote.InvalidAssociation: Invalid address:
> akka.tcp://sparkMaster@localhost:7077
>
> My submit statement looks like this,
>
> ./spark-submit --jars
> /Users/hadoop/kafka-0.8.1.1-src/core/build/libs/kafka_2.8.0-0.8.1.1.jar
> --class KafkaMain --master spark://localhost:7077 --deploy-mode cluster
> --num-executors 100 --driver-memory 1g --executor-memory 1g
> /Users/hadoop/dev/kafkahbase/kafka/src/sparkhbase.jar
>
>