You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sameer Tilak <ss...@live.com> on 2014/07/08 20:52:54 UTC
CoarseGrainedExecutorBackend: Driver Disassociated
Dear All,
When I look inside the following directory on my worker node:$SPARK_HOME/work/app-20140708110707-0001/3
I see the following error message:
log4j:WARN No appenders could be found for logger (org.apache.hadoop.conf.Configuration).log4j:WARN Please initialize the log4j system properly.log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.14/07/08 11:07:11 INFO SparkHadoopUtil: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties14/07/08 11:07:11 INFO SecurityManager: Changing view acls to: p52944414/07/08 11:07:11 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(p529444)14/07/08 11:07:12 INFO Slf4jLogger: Slf4jLogger started14/07/08 11:07:12 INFO Remoting: Starting remoting14/07/08 11:07:13 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkExecutor@pzxnvm2022.dcld.pldc.kp.org:34679]14/07/08 11:07:13 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkExecutor@pzxnvm2022.x.y.name.org:34679]14/07/08 11:07:13 INFO CoarseGrainedExecutorBackend: Connecting to driver: akka.tcp://spark@localhost:39701/user/CoarseGrainedScheduler14/07/08 11:07:13 INFO WorkerWatcher: Connecting to worker akka.tcp://sparkWorker@pzxnvm2022.x.y.name.org:37054/user/Worker14/07/08 11:07:13 ERROR CoarseGrainedExecutorBackend: Driver Disassociated [akka.tcp://sparkExecutor@pzxnvm2022.dcld.pldc.kp.org:34679] -> [akka
I am not sure what the problem is but it is preventing me to get the 4 node test cluster up and running.
Re: CoarseGrainedExecutorBackend: Driver Disassociated
Posted by Aaron Davidson <il...@gmail.com>.
By the way, you can run the sc.getConf.get("spark.driver.host") thing
inside spark-shell, whether or not the Executors actually start up
successfully.
On Tue, Jul 8, 2014 at 8:23 PM, Aaron Davidson <il...@gmail.com> wrote:
> You actually should avoid setting SPARK_PUBLIC_DNS unless necessary, I
> thought you might have preemptively done so. I think the issue is actually
> related to your network configuration, as Spark probably failed to find
> your driver's ip address. Do you see a warning on the driver that looks
> something like "Your hostname, localhost resolves to a loopback address,
> but we couldn't find any external IP address"?
>
> Either way, let's try to set SPARK_LOCAL_IP (see
> http://spark.apache.org/docs/latest/configuration.html) inside
> ~/spark/conf/spark-env.sh on your driver machine to an IP address that's
> reachable by your executors. Something like
>
> export SPARK_LOCAL_IP=194.168.1.105
>
> You can make sure it was set correctly by running
> sc.getConf.get("spark.driver.host"), which should return the driver
> hostname, and NOT "localhost".
>
> (Note that it's also possible that your /etc/hosts file contains a mapping
> from the driver's ip address to localhost, which it should not.)
>
>
> On Tue, Jul 8, 2014 at 2:23 PM, Sameer Tilak <ss...@live.com> wrote:
>
>> Hi Aaron,
>> Would really appreciate your help if you can point me to the
>> documentation. Is this something that I need to do with /etc/hosts on each
>> of the worker machines ? Or do I set SPARK_PUBLIC_DNS (if yes, what is the
>> format?) or something else?
>>
>> I have the following set up:
>>
>> master node: pzxnvm2018.x.y.org
>> worker nodes: pzxnvm2022.x.y.org pzxnvm2023.x.y.org pzxnvm2024.x.y.org
>>
>>
>> From: ilikerps@gmail.com
>> Date: Tue, 8 Jul 2014 11:59:54 -0700
>>
>> Subject: Re: CoarseGrainedExecutorBackend: Driver Disassociated
>> To: user@spark.apache.org
>>
>>
>> Hmm, looks like the Executor is trying to connect to the driver on
>> localhost, from this line:
>> 14/07/08 11:07:13 INFO CoarseGrainedExecutorBackend: Connecting to
>> driver: akka.tcp://spark@localhost:39701/user/CoarseGrainedScheduler
>>
>> What is your setup? Standalone mode with 4 separate machines? Are you
>> configuring the driver public dns name somewhere?
>>
>>
>> On Tue, Jul 8, 2014 at 11:52 AM, Sameer Tilak <ss...@live.com> wrote:
>>
>> Dear All,
>>
>> When I look inside the following directory on my worker node:
>> $SPARK_HOME/work/app-20140708110707-0001/3
>>
>> I see the following error message:
>>
>> log4j:WARN No appenders could be found for logger
>> (org.apache.hadoop.conf.Configuration).
>> log4j:WARN Please initialize the log4j system properly.
>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
>> more info.
>> 14/07/08 11:07:11 INFO SparkHadoopUtil: Using Spark's default log4j
>> profile: org/apache/spark/log4j-defaults.properties
>> 14/07/08 11:07:11 INFO SecurityManager: Changing view acls to: p529444
>> 14/07/08 11:07:11 INFO SecurityManager: SecurityManager: authentication
>> disabled; ui acls disabled; users with view permissions: Set(p529444)
>> 14/07/08 11:07:12 INFO Slf4jLogger: Slf4jLogger started
>> 14/07/08 11:07:12 INFO Remoting: Starting remoting
>> 14/07/08 11:07:13 INFO Remoting: Remoting started; listening on addresses
>> :[akka.tcp://sparkExecutor@pzxnvm2022.dcld.pldc.kp.org:34679
>> <http://pzxnvm2022.dcld.pldc.kp.org:34679>]
>> 14/07/08 11:07:13 INFO Remoting: Remoting now listens on addresses:
>> [akka.tcp://sparkExecutor@pzxnvm2022.x.y.name.org:34679
>> <http://pzxnvm2022.x.y.name.org:34679>]
>> 14/07/08 11:07:13 INFO CoarseGrainedExecutorBackend: Connecting to
>> driver: akka.tcp://spark@localhost:39701/user/CoarseGrainedScheduler
>> 14/07/08 11:07:13 INFO WorkerWatcher: Connecting to worker akka.tcp://
>> sparkWorker@pzxnvm2022.x.y.name.org:37054/user/Worker
>> <http://pzxnvm2022.x.y.name.org:37054/user/Worker>
>> 14/07/08 11:07:13 ERROR CoarseGrainedExecutorBackend: Driver
>> Disassociated [akka.tcp://sparkExecutor@pzxnvm2022.dcld.pldc.kp.org:34679
>> <http://pzxnvm2022.dcld.pldc.kp.org:34679>] -> [akka
>>
>>
>> I am not sure what the problem is but it is preventing me to get the 4
>> node test cluster up and running.
>>
>>
>>
>
Re: CoarseGrainedExecutorBackend: Driver Disassociated
Posted by Aaron Davidson <il...@gmail.com>.
You actually should avoid setting SPARK_PUBLIC_DNS unless necessary, I
thought you might have preemptively done so. I think the issue is actually
related to your network configuration, as Spark probably failed to find
your driver's ip address. Do you see a warning on the driver that looks
something like "Your hostname, localhost resolves to a loopback address,
but we couldn't find any external IP address"?
Either way, let's try to set SPARK_LOCAL_IP (see
http://spark.apache.org/docs/latest/configuration.html) inside
~/spark/conf/spark-env.sh on your driver machine to an IP address that's
reachable by your executors. Something like
export SPARK_LOCAL_IP=194.168.1.105
You can make sure it was set correctly by running
sc.getConf.get("spark.driver.host"), which should return the driver
hostname, and NOT "localhost".
(Note that it's also possible that your /etc/hosts file contains a mapping
from the driver's ip address to localhost, which it should not.)
On Tue, Jul 8, 2014 at 2:23 PM, Sameer Tilak <ss...@live.com> wrote:
> Hi Aaron,
> Would really appreciate your help if you can point me to the
> documentation. Is this something that I need to do with /etc/hosts on each
> of the worker machines ? Or do I set SPARK_PUBLIC_DNS (if yes, what is the
> format?) or something else?
>
> I have the following set up:
>
> master node: pzxnvm2018.x.y.org
> worker nodes: pzxnvm2022.x.y.org pzxnvm2023.x.y.org pzxnvm2024.x.y.org
>
>
> From: ilikerps@gmail.com
> Date: Tue, 8 Jul 2014 11:59:54 -0700
>
> Subject: Re: CoarseGrainedExecutorBackend: Driver Disassociated
> To: user@spark.apache.org
>
>
> Hmm, looks like the Executor is trying to connect to the driver on
> localhost, from this line:
> 14/07/08 11:07:13 INFO CoarseGrainedExecutorBackend: Connecting to driver:
> akka.tcp://spark@localhost:39701/user/CoarseGrainedScheduler
>
> What is your setup? Standalone mode with 4 separate machines? Are you
> configuring the driver public dns name somewhere?
>
>
> On Tue, Jul 8, 2014 at 11:52 AM, Sameer Tilak <ss...@live.com> wrote:
>
> Dear All,
>
> When I look inside the following directory on my worker node:
> $SPARK_HOME/work/app-20140708110707-0001/3
>
> I see the following error message:
>
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.conf.Configuration).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
> 14/07/08 11:07:11 INFO SparkHadoopUtil: Using Spark's default log4j
> profile: org/apache/spark/log4j-defaults.properties
> 14/07/08 11:07:11 INFO SecurityManager: Changing view acls to: p529444
> 14/07/08 11:07:11 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(p529444)
> 14/07/08 11:07:12 INFO Slf4jLogger: Slf4jLogger started
> 14/07/08 11:07:12 INFO Remoting: Starting remoting
> 14/07/08 11:07:13 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkExecutor@pzxnvm2022.dcld.pldc.kp.org:34679
> <http://pzxnvm2022.dcld.pldc.kp.org:34679>]
> 14/07/08 11:07:13 INFO Remoting: Remoting now listens on addresses:
> [akka.tcp://sparkExecutor@pzxnvm2022.x.y.name.org:34679
> <http://pzxnvm2022.x.y.name.org:34679>]
> 14/07/08 11:07:13 INFO CoarseGrainedExecutorBackend: Connecting to driver:
> akka.tcp://spark@localhost:39701/user/CoarseGrainedScheduler
> 14/07/08 11:07:13 INFO WorkerWatcher: Connecting to worker akka.tcp://
> sparkWorker@pzxnvm2022.x.y.name.org:37054/user/Worker
> <http://pzxnvm2022.x.y.name.org:37054/user/Worker>
> 14/07/08 11:07:13 ERROR CoarseGrainedExecutorBackend: Driver Disassociated
> [akka.tcp://sparkExecutor@pzxnvm2022.dcld.pldc.kp.org:34679
> <http://pzxnvm2022.dcld.pldc.kp.org:34679>] -> [akka
>
>
> I am not sure what the problem is but it is preventing me to get the 4
> node test cluster up and running.
>
>
>
RE: CoarseGrainedExecutorBackend: Driver Disassociated
Posted by Sameer Tilak <ss...@live.com>.
Hi Aaron,Would really appreciate your help if you can point me to the documentation. Is this something that I need to do with /etc/hosts on each of the worker machines ? Or do I set SPARK_PUBLIC_DNS (if yes, what is the format?) or something else?
I have the following set up:
master node: pzxnvm2018.x.y.orgworker nodes: pzxnvm2022.x.y.org pzxnvm2023.x.y.org pzxnvm2024.x.y.org
From: ilikerps@gmail.com
Date: Tue, 8 Jul 2014 11:59:54 -0700
Subject: Re: CoarseGrainedExecutorBackend: Driver Disassociated
To: user@spark.apache.org
Hmm, looks like the Executor is trying to connect to the driver on localhost, from this line:14/07/08 11:07:13 INFO CoarseGrainedExecutorBackend: Connecting to driver: akka.tcp://spark@localhost:39701/user/CoarseGrainedScheduler
What is your setup? Standalone mode with 4 separate machines? Are you configuring the driver public dns name somewhere?
On Tue, Jul 8, 2014 at 11:52 AM, Sameer Tilak <ss...@live.com> wrote:
Dear All,
When I look inside the following directory on my worker node:$SPARK_HOME/work/app-20140708110707-0001/3
I see the following error message:
log4j:WARN No appenders could be found for logger (org.apache.hadoop.conf.Configuration).log4j:WARN Please initialize the log4j system properly.log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
14/07/08 11:07:11 INFO SparkHadoopUtil: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties14/07/08 11:07:11 INFO SecurityManager: Changing view acls to: p529444
14/07/08 11:07:11 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(p529444)
14/07/08 11:07:12 INFO Slf4jLogger: Slf4jLogger started14/07/08 11:07:12 INFO Remoting: Starting remoting14/07/08 11:07:13 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkExecutor@pzxnvm2022.dcld.pldc.kp.org:34679]
14/07/08 11:07:13 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkExecutor@pzxnvm2022.x.y.name.org:34679]
14/07/08 11:07:13 INFO CoarseGrainedExecutorBackend: Connecting to driver: akka.tcp://spark@localhost:39701/user/CoarseGrainedScheduler14/07/08 11:07:13 INFO WorkerWatcher: Connecting to worker akka.tcp://sparkWorker@pzxnvm2022.x.y.name.org:37054/user/Worker
14/07/08 11:07:13 ERROR CoarseGrainedExecutorBackend: Driver Disassociated [akka.tcp://sparkExecutor@pzxnvm2022.dcld.pldc.kp.org:34679] -> [akka
I am not sure what the problem is but it is preventing me to get the 4 node test cluster up and running.
Re: CoarseGrainedExecutorBackend: Driver Disassociated
Posted by Aaron Davidson <il...@gmail.com>.
Hmm, looks like the Executor is trying to connect to the driver on
localhost, from this line:
14/07/08 11:07:13 INFO CoarseGrainedExecutorBackend: Connecting to driver:
akka.tcp://spark@localhost:39701/user/CoarseGrainedScheduler
What is your setup? Standalone mode with 4 separate machines? Are you
configuring the driver public dns name somewhere?
On Tue, Jul 8, 2014 at 11:52 AM, Sameer Tilak <ss...@live.com> wrote:
> Dear All,
>
> When I look inside the following directory on my worker node:
> $SPARK_HOME/work/app-20140708110707-0001/3
>
> I see the following error message:
>
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.conf.Configuration).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
> 14/07/08 11:07:11 INFO SparkHadoopUtil: Using Spark's default log4j
> profile: org/apache/spark/log4j-defaults.properties
> 14/07/08 11:07:11 INFO SecurityManager: Changing view acls to: p529444
> 14/07/08 11:07:11 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(p529444)
> 14/07/08 11:07:12 INFO Slf4jLogger: Slf4jLogger started
> 14/07/08 11:07:12 INFO Remoting: Starting remoting
> 14/07/08 11:07:13 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkExecutor@pzxnvm2022.dcld.pldc.kp.org:34679]
> 14/07/08 11:07:13 INFO Remoting: Remoting now listens on addresses:
> [akka.tcp://sparkExecutor@pzxnvm2022.x.y.name.org:34679]
> 14/07/08 11:07:13 INFO CoarseGrainedExecutorBackend: Connecting to driver:
> akka.tcp://spark@localhost:39701/user/CoarseGrainedScheduler
> 14/07/08 11:07:13 INFO WorkerWatcher: Connecting to worker akka.tcp://
> sparkWorker@pzxnvm2022.x.y.name.org:37054/user/Worker
> 14/07/08 11:07:13 ERROR CoarseGrainedExecutorBackend: Driver Disassociated
> [akka.tcp://sparkExecutor@pzxnvm2022.dcld.pldc.kp.org:34679] -> [akka
>
>
> I am not sure what the problem is but it is preventing me to get the 4
> node test cluster up and running.
>
>