You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Amjad ALSHABANI <as...@gmail.com> on 2015/07/17 17:37:21 UTC
spark-shell with Yarn failed
Hello,
First of all I m a newbie in Spark ,
I m trying to start the spark-shell with yarn cluster by running:
$ spark-shell --master yarn-client
Sometimes it goes well, but most of the time I got an error:
Container exited with a non-zero exit code 10
Failing this attempt. Failing the application.
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1437145851944
final status: FAILED
tracking URL:
http://My-HadoopServer:50080/cluster/app/application_1437033338028_0030
user: hadoop
org.apache.spark.SparkException: Yarn application has already ended! It
might have been killed or unable to launch application master.
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:115)
....
....
....
searching in the yarn logs I got this log
$ yarn logs -applicationId application_1437033338028_0030
2015-07-17 17:11:03,961 - INFO
[sparkYarnAM-akka.actor.default-dispatcher-4:Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$3@74]
- Starting remoting
2015-07-17 17:11:04,200 - ERROR
[sparkYarnAM-akka.actor.default-dispatcher-4:Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$1@65]
- failed to bind to My-HadoopServer/10.98.105.11:0, shutting down Netty
transport
2015-07-17 17:11:04,210 - WARN [main:Logging$class@71] - Service
'sparkYarnAM' could not bind on port 0. Attempting port 1.
...
...
...
2015-07-17 17:11:05,123 - ERROR [main:Logging$class@96] - Uncaught
exception:
java.net.BindException: Failed to bind to:
My-HadoopServer/HadoopServerIP:0: Service 'sparkYarnAM' failed after 16
retries!
at
org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
at
akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:393)
at
akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:389)
at scala.util.Success$$anonfun$map$1.apply(Try.scala:206)
...
....
....
I m using Spark 1.3, Hadoop 2.6 ,
and in spark-env.sh it points to my hadoop configuration:
export HADOOP_CONF_DIR=/usr/hdp/2.2.4.4-16/hadoop/conf
Is this probleme coming from spark configuration or yarn configuration (or
spark with yarn confs)????
Any Ideas??
Amjad
Re: spark-shell with Yarn failed
Posted by Amjad ALSHABANI <as...@gmail.com>.
Are you running something on port 0 already?
No actually
I tired multiple ways to avoid this problem, and it seems to disappear when
I m setting the num-executors to 6 (My hadoop cluster is 3 nodes),
Could the num-executors have anything to do with the error I m getting??
On Sun, Jul 19, 2015 at 3:31 PM, ayan guha <gu...@gmail.com> wrote:
> Are you running something on port 0 already? Looks like spark app master
> is failing.
> On 19 Jul 2015 06:13, "Chester @work" <ch...@alpinenow.com> wrote:
>
>> it might be a network issue. The error states failed to bind the server
>> IP address
>>
>> Chester
>> Sent from my iPhone
>>
>> On Jul 18, 2015, at 11:46 AM, Amjad ALSHABANI <as...@gmail.com>
>> wrote:
>>
>> Does anybody have any idea about the error I m having.. I am really
>> clueless... And appreciate any idea :)
>>
>> Thanks in advance
>>
>> Amjad
>> On Jul 17, 2015 5:37 PM, "Amjad ALSHABANI" <as...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> First of all I m a newbie in Spark ,
>>>
>>> I m trying to start the spark-shell with yarn cluster by running:
>>>
>>> $ spark-shell --master yarn-client
>>>
>>> Sometimes it goes well, but most of the time I got an error:
>>>
>>> Container exited with a non-zero exit code 10
>>> Failing this attempt. Failing the application.
>>> ApplicationMaster host: N/A
>>> ApplicationMaster RPC port: -1
>>> queue: default
>>> start time: 1437145851944
>>> final status: FAILED
>>> tracking URL:
>>> http://My-HadoopServer:50080/cluster/app/application_1437033338028_0030
>>> user: hadoop
>>> org.apache.spark.SparkException: Yarn application has already ended! It
>>> might have been killed or unable to launch application master.
>>> at
>>> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:115)
>>> ....
>>> ....
>>> ....
>>>
>>> searching in the yarn logs I got this log
>>>
>>> $ yarn logs -applicationId application_1437033338028_0030
>>> 2015-07-17 17:11:03,961 - INFO
>>> [sparkYarnAM-akka.actor.default-dispatcher-4:Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$3@74]
>>> - Starting remoting
>>> 2015-07-17 17:11:04,200 - ERROR
>>> [sparkYarnAM-akka.actor.default-dispatcher-4:Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$1@65]
>>> - failed to bind to My-HadoopServer/10.98.105.11:0, shutting down Netty
>>> transport
>>> 2015-07-17 17:11:04,210 - WARN [main:Logging$class@71] - Service
>>> 'sparkYarnAM' could not bind on port 0. Attempting port 1.
>>> ...
>>> ...
>>> ...
>>> 2015-07-17 17:11:05,123 - ERROR [main:Logging$class@96] - Uncaught
>>> exception:
>>> java.net.BindException: Failed to bind to:
>>> My-HadoopServer/HadoopServerIP:0: Service 'sparkYarnAM' failed after 16
>>> retries!
>>> at
>>> org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
>>> at
>>> akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:393)
>>> at
>>> akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:389)
>>> at scala.util.Success$$anonfun$map$1.apply(Try.scala:206)
>>> ...
>>> ....
>>> ....
>>>
>>>
>>>
>>> I m using Spark 1.3, Hadoop 2.6 ,
>>>
>>> and in spark-env.sh it points to my hadoop configuration:
>>>
>>> export HADOOP_CONF_DIR=/usr/hdp/2.2.4.4-16/hadoop/conf
>>>
>>>
>>> Is this probleme coming from spark configuration or yarn configuration
>>> (or spark with yarn confs)????
>>>
>>> Any Ideas??
>>>
>>>
>>>
>>> Amjad
>>>
>>>
Re: spark-shell with Yarn failed
Posted by ayan guha <gu...@gmail.com>.
Are you running something on port 0 already? Looks like spark app master is
failing.
On 19 Jul 2015 06:13, "Chester @work" <ch...@alpinenow.com> wrote:
> it might be a network issue. The error states failed to bind the server IP
> address
>
> Chester
> Sent from my iPhone
>
> On Jul 18, 2015, at 11:46 AM, Amjad ALSHABANI <as...@gmail.com>
> wrote:
>
> Does anybody have any idea about the error I m having.. I am really
> clueless... And appreciate any idea :)
>
> Thanks in advance
>
> Amjad
> On Jul 17, 2015 5:37 PM, "Amjad ALSHABANI" <as...@gmail.com> wrote:
>
>> Hello,
>>
>> First of all I m a newbie in Spark ,
>>
>> I m trying to start the spark-shell with yarn cluster by running:
>>
>> $ spark-shell --master yarn-client
>>
>> Sometimes it goes well, but most of the time I got an error:
>>
>> Container exited with a non-zero exit code 10
>> Failing this attempt. Failing the application.
>> ApplicationMaster host: N/A
>> ApplicationMaster RPC port: -1
>> queue: default
>> start time: 1437145851944
>> final status: FAILED
>> tracking URL:
>> http://My-HadoopServer:50080/cluster/app/application_1437033338028_0030
>> user: hadoop
>> org.apache.spark.SparkException: Yarn application has already ended! It
>> might have been killed or unable to launch application master.
>> at
>> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:115)
>> ....
>> ....
>> ....
>>
>> searching in the yarn logs I got this log
>>
>> $ yarn logs -applicationId application_1437033338028_0030
>> 2015-07-17 17:11:03,961 - INFO
>> [sparkYarnAM-akka.actor.default-dispatcher-4:Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$3@74]
>> - Starting remoting
>> 2015-07-17 17:11:04,200 - ERROR
>> [sparkYarnAM-akka.actor.default-dispatcher-4:Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$1@65]
>> - failed to bind to My-HadoopServer/10.98.105.11:0, shutting down Netty
>> transport
>> 2015-07-17 17:11:04,210 - WARN [main:Logging$class@71] - Service
>> 'sparkYarnAM' could not bind on port 0. Attempting port 1.
>> ...
>> ...
>> ...
>> 2015-07-17 17:11:05,123 - ERROR [main:Logging$class@96] - Uncaught
>> exception:
>> java.net.BindException: Failed to bind to:
>> My-HadoopServer/HadoopServerIP:0: Service 'sparkYarnAM' failed after 16
>> retries!
>> at
>> org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
>> at
>> akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:393)
>> at
>> akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:389)
>> at scala.util.Success$$anonfun$map$1.apply(Try.scala:206)
>> ...
>> ....
>> ....
>>
>>
>>
>> I m using Spark 1.3, Hadoop 2.6 ,
>>
>> and in spark-env.sh it points to my hadoop configuration:
>>
>> export HADOOP_CONF_DIR=/usr/hdp/2.2.4.4-16/hadoop/conf
>>
>>
>> Is this probleme coming from spark configuration or yarn configuration
>> (or spark with yarn confs)????
>>
>> Any Ideas??
>>
>>
>>
>> Amjad
>>
>>
Re: spark-shell with Yarn failed
Posted by "Chester @work" <ch...@alpinenow.com>.
it might be a network issue. The error states failed to bind the server IP address
Chester
Sent from my iPhone
> On Jul 18, 2015, at 11:46 AM, Amjad ALSHABANI <as...@gmail.com> wrote:
>
> Does anybody have any idea about the error I m having.. I am really clueless... And appreciate any idea :)
>
> Thanks in advance
>
> Amjad
>
>> On Jul 17, 2015 5:37 PM, "Amjad ALSHABANI" <as...@gmail.com> wrote:
>> Hello,
>>
>> First of all I m a newbie in Spark ,
>>
>> I m trying to start the spark-shell with yarn cluster by running:
>>
>> $ spark-shell --master yarn-client
>>
>> Sometimes it goes well, but most of the time I got an error:
>>
>> Container exited with a non-zero exit code 10
>> Failing this attempt. Failing the application.
>> ApplicationMaster host: N/A
>> ApplicationMaster RPC port: -1
>> queue: default
>> start time: 1437145851944
>> final status: FAILED
>> tracking URL: http://My-HadoopServer:50080/cluster/app/application_1437033338028_0030
>> user: hadoop
>> org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
>> at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:115)
>> ....
>> ....
>> ....
>>
>> searching in the yarn logs I got this log
>>
>> $ yarn logs -applicationId application_1437033338028_0030
>> 2015-07-17 17:11:03,961 - INFO [sparkYarnAM-akka.actor.default-dispatcher-4:Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$3@74] - Starting remoting
>> 2015-07-17 17:11:04,200 - ERROR [sparkYarnAM-akka.actor.default-dispatcher-4:Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$1@65] - failed to bind to My-HadoopServer/10.98.105.11:0, shutting down Netty transport
>> 2015-07-17 17:11:04,210 - WARN [main:Logging$class@71] - Service 'sparkYarnAM' could not bind on port 0. Attempting port 1.
>> ...
>> ...
>> ...
>> 2015-07-17 17:11:05,123 - ERROR [main:Logging$class@96] - Uncaught exception:
>> java.net.BindException: Failed to bind to: My-HadoopServer/HadoopServerIP:0: Service 'sparkYarnAM' failed after 16 retries!
>> at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
>> at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:393)
>> at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:389)
>> at scala.util.Success$$anonfun$map$1.apply(Try.scala:206)
>> ...
>> ....
>> ....
>>
>>
>>
>> I m using Spark 1.3, Hadoop 2.6 ,
>>
>> and in spark-env.sh it points to my hadoop configuration:
>>
>> export HADOOP_CONF_DIR=/usr/hdp/2.2.4.4-16/hadoop/conf
>>
>>
>> Is this probleme coming from spark configuration or yarn configuration (or spark with yarn confs)????
>>
>> Any Ideas??
>>
>>
>>
>> Amjad
Re: spark-shell with Yarn failed
Posted by Amjad ALSHABANI <as...@gmail.com>.
Does anybody have any idea about the error I m having.. I am really
clueless... And appreciate any idea :)
Thanks in advance
Amjad
On Jul 17, 2015 5:37 PM, "Amjad ALSHABANI" <as...@gmail.com> wrote:
> Hello,
>
> First of all I m a newbie in Spark ,
>
> I m trying to start the spark-shell with yarn cluster by running:
>
> $ spark-shell --master yarn-client
>
> Sometimes it goes well, but most of the time I got an error:
>
> Container exited with a non-zero exit code 10
> Failing this attempt. Failing the application.
> ApplicationMaster host: N/A
> ApplicationMaster RPC port: -1
> queue: default
> start time: 1437145851944
> final status: FAILED
> tracking URL:
> http://My-HadoopServer:50080/cluster/app/application_1437033338028_0030
> user: hadoop
> org.apache.spark.SparkException: Yarn application has already ended! It
> might have been killed or unable to launch application master.
> at
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:115)
> ....
> ....
> ....
>
> searching in the yarn logs I got this log
>
> $ yarn logs -applicationId application_1437033338028_0030
> 2015-07-17 17:11:03,961 - INFO
> [sparkYarnAM-akka.actor.default-dispatcher-4:Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$3@74]
> - Starting remoting
> 2015-07-17 17:11:04,200 - ERROR
> [sparkYarnAM-akka.actor.default-dispatcher-4:Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$1@65]
> - failed to bind to My-HadoopServer/10.98.105.11:0, shutting down Netty
> transport
> 2015-07-17 17:11:04,210 - WARN [main:Logging$class@71] - Service
> 'sparkYarnAM' could not bind on port 0. Attempting port 1.
> ...
> ...
> ...
> 2015-07-17 17:11:05,123 - ERROR [main:Logging$class@96] - Uncaught
> exception:
> java.net.BindException: Failed to bind to:
> My-HadoopServer/HadoopServerIP:0: Service 'sparkYarnAM' failed after 16
> retries!
> at
> org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
> at
> akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:393)
> at
> akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:389)
> at scala.util.Success$$anonfun$map$1.apply(Try.scala:206)
> ...
> ....
> ....
>
>
>
> I m using Spark 1.3, Hadoop 2.6 ,
>
> and in spark-env.sh it points to my hadoop configuration:
>
> export HADOOP_CONF_DIR=/usr/hdp/2.2.4.4-16/hadoop/conf
>
>
> Is this probleme coming from spark configuration or yarn configuration (or
> spark with yarn confs)????
>
> Any Ideas??
>
>
>
> Amjad
>
>