You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Liangzhao Zeng <li...@gmail.com> on 2016/08/02 17:06:37 UTC
Job can not terminated in Spark 2.0 on Yarn
Hi,
I migrate my code to Spark 2.0 from 1.6. It finish last stage (and
result is correct) but get following errors then start over.
Any idea on what happen?
16/08/02 16:59:33 ERROR scheduler.LiveListenerBus: SparkListenerBus
has already stopped! Dropping event
SparkListenerExecutorMetricsUpdate(2,WrappedArray())
16/08/02 16:59:33 ERROR scheduler.LiveListenerBus: SparkListenerBus
has already stopped! Dropping event
SparkListenerExecutorMetricsUpdate(115,WrappedArray())
16/08/02 16:59:33 ERROR scheduler.LiveListenerBus: SparkListenerBus
has already stopped! Dropping event
SparkListenerExecutorMetricsUpdate(70,WrappedArray())
16/08/02 16:59:33 WARN yarn.YarnAllocator: Expected to find pending
requests, but found none.
16/08/02 16:59:33 WARN netty.Dispatcher: Message
RemoteProcessDisconnected(17.138.53.26:55338) dropped. Could not find
MapOutputTracker.
Cheers,
LZ
Re: Job can not terminated in Spark 2.0 on Yarn
Posted by Liangzhao Zeng <li...@gmail.com>.
It try to execute the job again, from the first stage.
发自我的 iPhone
> 在 Aug 2, 2016，11:24 AM，dhruve ashar <dh...@gmail.com> 写道：
>
> Hi LZ,
>
> Getting those error messages in logs is normal behavior. When the job completes, it shuts down the SparkListenerBus as there's no need of relaying any spark events to the interested registered listeners. So trying to add events from executors which are yet to shutdown, logs the error message because the event listener bus has stopped.
>
> "I migrate my code to Spark 2.0 from 1.6. It finish last stage (and result is correct) but get following errors then start over."
>
> Can you elaborate on what do you mean by start over?
>
> -Dhruve
>
>
>
>> On Tue, Aug 2, 2016 at 1:01 PM, Ted Yu <yu...@gmail.com> wrote:
>> Which hadoop version are you using ?
>>
>> Can you show snippet of your code ?
>>
>> Thanks
>>
>>> On Tue, Aug 2, 2016 at 10:06 AM, Liangzhao Zeng <li...@gmail.com> wrote:
>>> Hi,
>>>
>>> I migrate my code to Spark 2.0 from 1.6. It finish last stage (and result is correct) but get following errors then start over.
>>>
>>> Any idea on what happen?
>>>
>>> 16/08/02 16:59:33 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(2,WrappedArray())
>>> 16/08/02 16:59:33 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(115,WrappedArray())
>>> 16/08/02 16:59:33 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(70,WrappedArray())
>>> 16/08/02 16:59:33 WARN yarn.YarnAllocator: Expected to find pending requests, but found none.
>>> 16/08/02 16:59:33 WARN netty.Dispatcher: Message RemoteProcessDisconnected(17.138.53.26:55338) dropped. Could not find MapOutputTracker.
>>>
>>>
>>> Cheers,
>>>
>>> LZ
>
>
>
> --
> -Dhruve Ashar
>
Re: Job can not terminated in Spark 2.0 on Yarn
Posted by dhruve ashar <dh...@gmail.com>.
Hi LZ,
Getting those error messages in logs is normal behavior. When the job
completes, it shuts down the SparkListenerBus as there's no need of
relaying any spark events to the interested registered listeners. So trying
to add events from executors which are yet to shutdown, logs the error
message because the event listener bus has stopped.
"I migrate my code to Spark 2.0 from 1.6. It finish last stage (and
result is correct) but get following errors then start over."
Can you elaborate on what do you mean by start over?
-Dhruve
On Tue, Aug 2, 2016 at 1:01 PM, Ted Yu <yu...@gmail.com> wrote:
> Which hadoop version are you using ?
>
> Can you show snippet of your code ?
>
> Thanks
>
> On Tue, Aug 2, 2016 at 10:06 AM, Liangzhao Zeng <li...@gmail.com>
> wrote:
>
>> Hi,
>>
>>
>> I migrate my code to Spark 2.0 from 1.6. It finish last stage (and result is correct) but get following errors then start over.
>>
>>
>> Any idea on what happen?
>>
>>
>> 16/08/02 16:59:33 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(2,WrappedArray())
>> 16/08/02 16:59:33 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(115,WrappedArray())
>> 16/08/02 16:59:33 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(70,WrappedArray())
>> 16/08/02 16:59:33 WARN yarn.YarnAllocator: Expected to find pending requests, but found none.
>> 16/08/02 16:59:33 WARN netty.Dispatcher: Message RemoteProcessDisconnected(17.138.53.26:55338) dropped. Could not find MapOutputTracker.
>>
>>
>>
>> Cheers,
>>
>>
>> LZ
>>
>>
>
--
-Dhruve Ashar
Re: Job can not terminated in Spark 2.0 on Yarn
Posted by Liangzhao Zeng <li...@gmail.com>.
It works fine now after I remove System.exit(0)
�����ҵ� iPhone
> �� Aug 2, 2016��12:46 PM��dhruve ashar <dh...@gmail.com> ���
>
> Can you provide additional logs.
>
>> On Tue, Aug 2, 2016 at 2:13 PM, Liangzhao Zeng <li...@gmail.com> wrote:
>> It is 2.6 and code is very simple. I load data file from Hdfs to create rdd then same some samples.
>>
>>
>> Thanks
>>
>> �����ҵ� iPhone
>>
>>> �� Aug 2, 2016��11:01 AM��Ted Yu <yu...@gmail.com> ���
>>>
>>> Which hadoop version are you using ?
>>>
>>> Can you show snippet of your code ?
>>>
>>> Thanks
>>>
>>>> On Tue, Aug 2, 2016 at 10:06 AM, Liangzhao Zeng <li...@gmail.com> wrote:
>>>> Hi,
>>>>
>>>> I migrate my code to Spark 2.0 from 1.6. It finish last stage (and result is correct) but get following errors then start over.
>>>>
>>>> Any idea on what happen?
>>>>
>>>> 16/08/02 16:59:33 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(2,WrappedArray())
>>>> 16/08/02 16:59:33 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(115,WrappedArray())
>>>> 16/08/02 16:59:33 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(70,WrappedArray())
>>>> 16/08/02 16:59:33 WARN yarn.YarnAllocator: Expected to find pending requests, but found none.
>>>> 16/08/02 16:59:33 WARN netty.Dispatcher: Message RemoteProcessDisconnected(17.138.53.26:55338) dropped. Could not find MapOutputTracker.
>>>>
>>>>
>>>> Cheers,
>>>>
>>>> LZ
>
>
>
> --
> -Dhruve Ashar
>
Re: Job can not terminated in Spark 2.0 on Yarn
Posted by dhruve ashar <dh...@gmail.com>.
Can you provide additional logs.
On Tue, Aug 2, 2016 at 2:13 PM, Liangzhao Zeng <li...@gmail.com>
wrote:
> It is 2.6 and code is very simple. I load data file from Hdfs to create
> rdd then same some samples.
>
>
> Thanks
>
> 发自我的 iPhone
>
> 在 Aug 2, 2016，11:01 AM，Ted Yu <yu...@gmail.com> 写道：
>
> Which hadoop version are you using ?
>
> Can you show snippet of your code ?
>
> Thanks
>
> On Tue, Aug 2, 2016 at 10:06 AM, Liangzhao Zeng <li...@gmail.com>
> wrote:
>
>> Hi,
>>
>>
>> I migrate my code to Spark 2.0 from 1.6. It finish last stage (and result is correct) but get following errors then start over.
>>
>>
>> Any idea on what happen?
>>
>>
>> 16/08/02 16:59:33 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(2,WrappedArray())
>> 16/08/02 16:59:33 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(115,WrappedArray())
>> 16/08/02 16:59:33 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(70,WrappedArray())
>> 16/08/02 16:59:33 WARN yarn.YarnAllocator: Expected to find pending requests, but found none.
>> 16/08/02 16:59:33 WARN netty.Dispatcher: Message RemoteProcessDisconnected(17.138.53.26:55338) dropped. Could not find MapOutputTracker.
>>
>>
>>
>> Cheers,
>>
>>
>> LZ
>>
>>
>
--
-Dhruve Ashar
Re: Job can not terminated in Spark 2.0 on Yarn
Posted by Liangzhao Zeng <li...@gmail.com>.
It is 2.6 and code is very simple. I load data file from Hdfs to create rdd then same some samples.
Thanks
�����ҵ� iPhone
> �� Aug 2, 2016��11:01 AM��Ted Yu <yu...@gmail.com> ���
>
> Which hadoop version are you using ?
>
> Can you show snippet of your code ?
>
> Thanks
>
>> On Tue, Aug 2, 2016 at 10:06 AM, Liangzhao Zeng <li...@gmail.com> wrote:
>> Hi,
>>
>> I migrate my code to Spark 2.0 from 1.6. It finish last stage (and result is correct) but get following errors then start over.
>>
>> Any idea on what happen?
>>
>> 16/08/02 16:59:33 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(2,WrappedArray())
>> 16/08/02 16:59:33 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(115,WrappedArray())
>> 16/08/02 16:59:33 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(70,WrappedArray())
>> 16/08/02 16:59:33 WARN yarn.YarnAllocator: Expected to find pending requests, but found none.
>> 16/08/02 16:59:33 WARN netty.Dispatcher: Message RemoteProcessDisconnected(17.138.53.26:55338) dropped. Could not find MapOutputTracker.
>>
>>
>> Cheers,
>>
>> LZ
>
Re: Job can not terminated in Spark 2.0 on Yarn
Posted by Ted Yu <yu...@gmail.com>.
Which hadoop version are you using ?
Can you show snippet of your code ?
Thanks
On Tue, Aug 2, 2016 at 10:06 AM, Liangzhao Zeng <li...@gmail.com>
wrote:
> Hi,
>
>
> I migrate my code to Spark 2.0 from 1.6. It finish last stage (and result is correct) but get following errors then start over.
>
>
> Any idea on what happen?
>
>
> 16/08/02 16:59:33 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(2,WrappedArray())
> 16/08/02 16:59:33 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(115,WrappedArray())
> 16/08/02 16:59:33 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(70,WrappedArray())
> 16/08/02 16:59:33 WARN yarn.YarnAllocator: Expected to find pending requests, but found none.
> 16/08/02 16:59:33 WARN netty.Dispatcher: Message RemoteProcessDisconnected(17.138.53.26:55338) dropped. Could not find MapOutputTracker.
>
>
>
> Cheers,
>
>
> LZ
>
>