You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by zhangliyun <ke...@126.com> on 2020/05/07 02:14:37 UTC
a spark job hangs 20 hours and it shows 2 tasks not finished in
stage page but all tasks shows finished or failed in task page
Hi all:
i want to ask a question, it seems that my spark job hangs as 20+ hours.
in the spark history log, it shows 8999 completed task while 2 not finished. but when i go to the tasks page. i did not find any running tasks. All tasks are failed or Successful. I guess it seems that all tasks are finished while driver is not notified. i want to ask how to check or verify my guess.
Re: a spark job hangs 20 hours and it shows 2 tasks not finished in
stage page but all tasks shows finished or failed in task page
Posted by ML Books <ma...@gmail.com>.
I think this question might sound silly. How to set yarn queue to each
stage??
Whenever we set yarn queue to the application level. It doesn't reflect in
stages page. It always shows default.
On Thu, May 7, 2020, 9:23 AM angers.zhu <an...@gmail.com> wrote:
> hi zhangliyun,
>
> Loss event in ListenerBus?
>
> I meet this case when event queue’s capacity is too small.
>
> angers.zhu
> angers.zhu@gmail.com
>
> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=angers.zhu&uid=angers.zhu%40gmail.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22angers.zhu%40gmail.com%22%5D>
> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制
>
> On 05/7/2020 11:50，zhangliyun<ke...@126.com> <ke...@126.com> wrote：
>
>
>
> the stage shows it is running, no description , see attached picture.
>
>
>
>
> 在 2020-05-07 10:59:45，"ML Books" <ma...@gmail.com> 写道：
>
> You can click on the description of stage. It redirects to the tasks page.
> Please share that.
>
>
> On Thu, May 7, 2020, 7:44 AM zhangliyun <ke...@126.com> wrote:
>
>> Hi all:
>> i want to ask a question, it seems that my spark job hangs as 20+ hours.
>> in the spark history log, it shows 8999 completed task while 2 not
>> finished. but when i go to the tasks page. i did not find any running
>> tasks. All tasks are failed or Successful. I guess it seems that all
>> tasks are finished while driver is not notified. i want to ask how to
>> check or verify my guess.
>>
>>
>>
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
>
>
>
>
Re: Re: Re: a spark job hangs 20 hours and it shows 2 tasks not
finished in stage page but all tasks shows finished or failed in task page
Posted by Wenchen Fan <cl...@gmail.com>.
Is it a UI issue or your job does hang forever? You can check the driver
side log and see if the jobs were completed normally.
On Mon, May 11, 2020 at 5:35 AM zhangliyun <ke...@126.com> wrote:
>
> spark 2.3.1
>
>
>
>
>
> At 2020-05-08 16:53:01, "Wenchen Fan" <cl...@gmail.com> wrote:
>
> Which Spark version do you use? If an event is so important that missing
> it would hang the application forever, we should not let it happen or fail
> earlier to give a better error message.
>
> On Thu, May 7, 2020 at 3:37 PM zhangliyun <ke...@126.com> wrote:
>
>> thanks all, I grep
>>
>> grep "Dropping event from queue" bslv3.stdout
>>
>> 2020-05-05 16:00:39,084 ERROR AsyncEventQueue: Dropping event from queue
>> executorManagement. This likely means one of the listeners is too slow and
>> cannot keep up with the rate at which tasks are being started by the
>> scheduler.
>>
>> I saw the default value for spark.scheduler.listenerbus.eventqueue.capacity
>> is 10000, so what I need to do is to double it?
>> like 20000. is there any bad effect if enlarge it to 20000?
>>
>> Best Regards
>> Kelly Zhang
>>
>> 在 2020-05-07 12:04:06，"Sandeep Katta" <sa...@gmail.com>
>> 写道：
>>
>> Yes we too faced this problem, just try to grep ‘ Dropping event from
>> queue’ in the driver logs
>>
>> If you find this then it is problem with the event queue capacity, just
>> try to increase spark.scheduler.listenerbus.eventqueue.capacity
>>
>> On Thu, 7 May 2020 at 9:24 AM, angers.zhu <an...@gmail.com> wrote:
>>
>>> hi zhangliyun,
>>>
>>> Loss event in ListenerBus?
>>>
>>> I meet this case when event queue’s capacity is too small.
>>>
>>> angers.zhu
>>> angers.zhu@gmail.com
>>>
>>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=angers.zhu&uid=angers.zhu%40gmail.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22angers.zhu%40gmail.com%22%5D>
>>>
>>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=angers.zhu&uid=angers.zhu%40gmail.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22angers.zhu%40gmail.com%22%5D>
>>> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制
>>>
>>> On 05/7/2020 11:50，zhangliyun<ke...@126.com> <ke...@126.com>
>>> wrote：
>>>
>>>
>>>
>>> the stage shows it is running, no description , see attached picture.
>>>
>>>
>>>
>>>
>>> 在 2020-05-07 10:59:45，"ML Books" <ma...@gmail.com> 写道：
>>>
>>> You can click on the description of stage. It redirects to the tasks
>>> page. Please share that.
>>>
>>>
>>> On Thu, May 7, 2020, 7:44 AM zhangliyun <ke...@126.com> wrote:
>>>
>>>> Hi all:
>>>> i want to ask a question, it seems that my spark job hangs as 20+
>>>> hours.
>>>> in the spark history log, it shows 8999 completed task while 2 not
>>>> finished. but when i go to the tasks page. i did not find any running
>>>> tasks. All tasks are failed or Successful. I guess it seems that all
>>>> tasks are finished while driver is not notified. i want to ask how to
>>>> check or verify my guess.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>
Re:Re: Re: a spark job hangs 20 hours and it shows 2 tasks not
finished in stage page but all tasks shows finished or failed in task page
Posted by zhangliyun <ke...@126.com>.
spark 2.3.1
At 2020-05-08 16:53:01, "Wenchen Fan" <cl...@gmail.com> wrote:
Which Spark version do you use? If an event is so important that missing it would hang the application forever, we should not let it happen or fail earlier to give a better error message.
On Thu, May 7, 2020 at 3:37 PM zhangliyun <ke...@126.com> wrote:
thanks all, I grep
grep "Dropping event from queue" bslv3.stdout
2020-05-05 16:00:39,084 ERROR AsyncEventQueue: Dropping event from queue executorManagement. This likely means one of the listeners is too slow and cannot keep up with the rate at which tasks are being started by the scheduler.
I saw the default value for spark.scheduler.listenerbus.eventqueue.capacity is 10000, so what I need to do is to double it?
like 20000. is there any bad effect if enlarge it to 20000?
Best Regards
Kelly Zhang
在 2020-05-07 12:04:06，"Sandeep Katta" <sa...@gmail.com> 写道：
Yes we too faced this problem, just try to grep ‘ Dropping event from queue’ in the driver logs
If you find this then it is problem with the event queue capacity, just try to increase spark.scheduler.listenerbus.eventqueue.capacity
On Thu, 7 May 2020 at 9:24 AM, angers.zhu <an...@gmail.com> wrote:
hi zhangliyun,
Loss event in ListenerBus?
I meet this case when event queue’s capacity is too small.
| |
angers.zhu
|
|
angers.zhu@gmail.com
|
签名由网易邮箱大师定制
On 05/7/2020 11:50，zhangliyun<ke...@126.com> wrote：
the stage shows it is running, no description , see attached picture.
在 2020-05-07 10:59:45，"ML Books" <ma...@gmail.com> 写道：
You can click on the description of stage. It redirects to the tasks page. Please share that.
On Thu, May 7, 2020, 7:44 AM zhangliyun <ke...@126.com> wrote:
Hi all:
i want to ask a question, it seems that my spark job hangs as 20+ hours.
in the spark history log, it shows 8999 completed task while 2 not finished. but when i go to the tasks page. i did not find any running tasks. All tasks are failed or Successful. I guess it seems that all tasks are finished while driver is not notified. i want to ask how to check or verify my guess.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Re: a spark job hangs 20 hours and it shows 2 tasks not finished
in stage page but all tasks shows finished or failed in task page
Posted by Wenchen Fan <cl...@gmail.com>.
Which Spark version do you use? If an event is so important that missing it
would hang the application forever, we should not let it happen or fail
earlier to give a better error message.
On Thu, May 7, 2020 at 3:37 PM zhangliyun <ke...@126.com> wrote:
> thanks all, I grep
>
> grep "Dropping event from queue" bslv3.stdout
>
> 2020-05-05 16:00:39,084 ERROR AsyncEventQueue: Dropping event from queue
> executorManagement. This likely means one of the listeners is too slow and
> cannot keep up with the rate at which tasks are being started by the
> scheduler.
>
> I saw the default value for spark.scheduler.listenerbus.eventqueue.capacity
> is 10000, so what I need to do is to double it?
> like 20000. is there any bad effect if enlarge it to 20000?
>
> Best Regards
> Kelly Zhang
>
> 在 2020-05-07 12:04:06，"Sandeep Katta" <sa...@gmail.com>
> 写道：
>
> Yes we too faced this problem, just try to grep ‘ Dropping event from
> queue’ in the driver logs
>
> If you find this then it is problem with the event queue capacity, just
> try to increase spark.scheduler.listenerbus.eventqueue.capacity
>
> On Thu, 7 May 2020 at 9:24 AM, angers.zhu <an...@gmail.com> wrote:
>
>> hi zhangliyun,
>>
>> Loss event in ListenerBus?
>>
>> I meet this case when event queue’s capacity is too small.
>>
>> angers.zhu
>> angers.zhu@gmail.com
>>
>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=angers.zhu&uid=angers.zhu%40gmail.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22angers.zhu%40gmail.com%22%5D>
>>
>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=angers.zhu&uid=angers.zhu%40gmail.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22angers.zhu%40gmail.com%22%5D>
>> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制
>>
>> On 05/7/2020 11:50，zhangliyun<ke...@126.com> <ke...@126.com>
>> wrote：
>>
>>
>>
>> the stage shows it is running, no description , see attached picture.
>>
>>
>>
>>
>> 在 2020-05-07 10:59:45，"ML Books" <ma...@gmail.com> 写道：
>>
>> You can click on the description of stage. It redirects to the tasks
>> page. Please share that.
>>
>>
>> On Thu, May 7, 2020, 7:44 AM zhangliyun <ke...@126.com> wrote:
>>
>>> Hi all:
>>> i want to ask a question, it seems that my spark job hangs as 20+
>>> hours.
>>> in the spark history log, it shows 8999 completed task while 2 not
>>> finished. but when i go to the tasks page. i did not find any running
>>> tasks. All tasks are failed or Successful. I guess it seems that all
>>> tasks are finished while driver is not notified. i want to ask how to
>>> check or verify my guess.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
>>
>>
>>
>>
>
>
>
Re:Re: a spark job hangs 20 hours and it shows 2 tasks not finished
in stage page but all tasks shows finished or failed in task page
Posted by zhangliyun <ke...@126.com>.
thanks all, I grep
grep "Dropping event from queue" bslv3.stdout
2020-05-05 16:00:39,084 ERROR AsyncEventQueue: Dropping event from queue executorManagement. This likely means one of the listeners is too slow and cannot keep up with the rate at which tasks are being started by the scheduler.
I saw the default value for spark.scheduler.listenerbus.eventqueue.capacity is 10000, so what I need to do is to double it?
like 20000. is there any bad effect if enlarge it to 20000?
Best Regards
Kelly Zhang
在 2020-05-07 12:04:06，"Sandeep Katta" <sa...@gmail.com> 写道：
Yes we too faced this problem, just try to grep ‘ Dropping event from queue’ in the driver logs
If you find this then it is problem with the event queue capacity, just try to increase spark.scheduler.listenerbus.eventqueue.capacity
On Thu, 7 May 2020 at 9:24 AM, angers.zhu <an...@gmail.com> wrote:
hi zhangliyun,
Loss event in ListenerBus?
I meet this case when event queue’s capacity is too small.
| |
angers.zhu
|
|
angers.zhu@gmail.com
|
签名由网易邮箱大师定制
On 05/7/2020 11:50，zhangliyun<ke...@126.com> wrote：
the stage shows it is running, no description , see attached picture.
在 2020-05-07 10:59:45，"ML Books" <ma...@gmail.com> 写道：
You can click on the description of stage. It redirects to the tasks page. Please share that.
On Thu, May 7, 2020, 7:44 AM zhangliyun <ke...@126.com> wrote:
Hi all:
i want to ask a question, it seems that my spark job hangs as 20+ hours.
in the spark history log, it shows 8999 completed task while 2 not finished. but when i go to the tasks page. i did not find any running tasks. All tasks are failed or Successful. I guess it seems that all tasks are finished while driver is not notified. i want to ask how to check or verify my guess.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: a spark job hangs 20 hours and it shows 2 tasks not finished in
stage page but all tasks shows finished or failed in task page
Posted by Sandeep Katta <sa...@gmail.com>.
Yes we too faced this problem, just try to grep ‘ Dropping event from
queue’ in the driver logs
If you find this then it is problem with the event queue capacity, just try
to increase spark.scheduler.listenerbus.eventqueue.capacity
On Thu, 7 May 2020 at 9:24 AM, angers.zhu <an...@gmail.com> wrote:
> hi zhangliyun,
>
> Loss event in ListenerBus?
>
> I meet this case when event queue’s capacity is too small.
>
> angers.zhu
> angers.zhu@gmail.com
>
> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=angers.zhu&uid=angers.zhu%40gmail.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22angers.zhu%40gmail.com%22%5D>
>
> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=angers.zhu&uid=angers.zhu%40gmail.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22angers.zhu%40gmail.com%22%5D>
> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制
>
> On 05/7/2020 11:50，zhangliyun<ke...@126.com> <ke...@126.com> wrote：
>
>
>
> the stage shows it is running, no description , see attached picture.
>
>
>
>
> 在 2020-05-07 10:59:45，"ML Books" <ma...@gmail.com> 写道：
>
> You can click on the description of stage. It redirects to the tasks page.
> Please share that.
>
>
> On Thu, May 7, 2020, 7:44 AM zhangliyun <ke...@126.com> wrote:
>
>> Hi all:
>> i want to ask a question, it seems that my spark job hangs as 20+ hours.
>> in the spark history log, it shows 8999 completed task while 2 not
>> finished. but when i go to the tasks page. i did not find any running
>> tasks. All tasks are failed or Successful. I guess it seems that all
>> tasks are finished while driver is not notified. i want to ask how to
>> check or verify my guess.
>>
>>
>>
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
>
>
>
>
Re: a spark job hangs 20 hours and it shows 2 tasks not finished in stage page but all tasks shows finished or failed in task page
Posted by "angers.zhu" <an...@gmail.com>.
hi zhangliyun,
Loss event in ListenerBus?
I meet this case when event queue’s capacity is too small.
[  |
angers.zhu
---|---
angers.zhu@gmail.com
](https://maas.mail.163.com/dashi-web-
extend/html/proSignature.html?ftlId=1&name=angers.zhu&uid=angers.zhu%40gmail.com&iconUrl=https%3A%2F%2Fmail-
online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22angers.zhu%40gmail.com%22%5D)
签名由 [网易邮箱大师](https://mail.163.com/dashi/dlpro.html?from=mail81) 定制
On 05/7/2020 11:50，[zhangliyun<ke...@126.com>](mailto:kellyzly@126.com)
wrote：
>
>
>
>
> the stage shows it is running, no description , see attached
picture.
>
>
>
>
>
>
>
>
>
>
> 在 2020-05-07 10:59:45，"ML Books" <ma...@gmail.com> 写道：
>
>> You can click on the description of stage. It redirects to the tasks page.
Please share that.
>>
>>
>
>>
>>
>
>>
>> On Thu, May 7, 2020, 7:44 AM zhangliyun
<[kellyzly@126.com](mailto:kellyzly@126.com)> wrote:
>
>>
>>> Hi all:
>>>
>>> i want to ask a question, it seems that my spark job hangs as 20+ hours.
>>>
>>> in the spark history log, it shows 8999 completed task while 2 not
finished. but when i go to the tasks page. i did not find any running tasks.
All tasks are failed or Successful. I guess it seems that all tasks are
finished while driver is not notified. i want to ask how to check or verify
my guess.
>>>
>>>
>
>>>
>>>
>
>
>>>
>>>
>>>
>>>
>
>
>>>
>>>
>>>
>>>
> \---------------------------------------------------------------------
> To unsubscribe e-mail: [dev-unsubscribe@spark.apache.org](mailto:dev-
unsubscribe@spark.apache.org)
>
>
>
>
>
>
Re:Re: a spark job hangs 20 hours and it shows 2 tasks not finished
in stage page but all tasks shows finished or failed in task page
Posted by zhangliyun <ke...@126.com>.
the stage shows it is running, no description , see attached picture.
�� 2020-05-07 10:59:45��"ML Books" <ma...@gmail.com> ���
You can click on the description of stage. It redirects to the tasks page. Please share that.
On Thu, May 7, 2020, 7:44 AM zhangliyun <ke...@126.com> wrote:
Hi all:
i want to ask a question, it seems that my spark job hangs as 20+ hours.
in the spark history log, it shows 8999 completed task while 2 not finished. but when i go to the tasks page. i did not find any running tasks. All tasks are failed or Successful. I guess it seems that all tasks are finished while driver is not notified. i want to ask how to check or verify my guess.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: a spark job hangs 20 hours and it shows 2 tasks not finished in
stage page but all tasks shows finished or failed in task page
Posted by ML Books <ma...@gmail.com>.
You can click on the description of stage. It redirects to the tasks page.
Please share that.
On Thu, May 7, 2020, 7:44 AM zhangliyun <ke...@126.com> wrote:
> Hi all:
> i want to ask a question, it seems that my spark job hangs as 20+ hours.
> in the spark history log, it shows 8999 completed task while 2 not
> finished. but when i go to the tasks page. i did not find any running
> tasks. All tasks are failed or Successful. I guess it seems that all
> tasks are finished while driver is not notified. i want to ask how to
> check or verify my guess.
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org