You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Aakash Basu <aa...@gmail.com> on 2018/05/27 07:45:54 UTC
Spark AsyncEventQueue doubt
Hi,
I'm getting the below ERROR and WARN when running a little heavy
calculation on a dataset -
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
> setLogLevel(newLevel).
> 2018-05-27 12:51:11 ERROR AsyncEventQueue:70 - Dropping event from queue
> appStatus. This likely means one of the listeners is too slow and cannot
> keep up with the rate at which tasks are being started by the scheduler.
> 2018-05-27 12:51:11 WARN AsyncEventQueue:66 - Dropped
> com.codahale.metrics.Counter@1d8423d1 events from appStatus since Thu Jan
> 01 05:30:00 IST 1970.
> [Stage 7784:>(9 + 3) / 34][Stage 7786:(53 + 38) / 200][Stage 7789:(74 +
> 53) / 200]2018-05-27 12:52:14 WARN AsyncEventQueue:66 - Dropped
> com.codahale.metrics.Counter@1d8423d1 events from appStatus since Sun May
> 27 12:51:11 IST 2018.
> [Stage 7784:>(9 + 3) / 34][Stage 7786:(53 + 38) / 200][Stage 7789:(74 +
> 53) / 200]2018-05-27 12:53:14 WARN AsyncEventQueue:66 - Dropped
> com.codahale.metrics.Counter@1d8423d1 events from appStatus since Sun May
> 27 12:52:14 IST 2018.
> 2018-05-27 12:54:14 WARN AsyncEventQueue:66 - Dropped
> com.codahale.metrics.Counter@1d8423d1 events from appStatus since Sun May
> 27 12:53:14 IST 2018.
> [Stage 7784:>(9 + 3) / 34][Stage 7786:(53 + 38) / 200][Stage 7789:(74 +
> 53) / 200]2018-05-27 12:55:14 WARN AsyncEventQueue:66 - Dropped
> com.codahale.metrics.Counter@1d8423d1 events from appStatus since Sun May
> 27 12:54:14 IST 2018.
> [Stage 7784:>(9 + 3) / 34][Stage 7786:(53 + 38) / 200][Stage 7789:(74 +
> 53) / 200]2018-05-27 12:56:15 WARN AsyncEventQueue:66 - Dropped
> com.codahale.metrics.Counter@1d8423d1 events from appStatus since Sun May
> 27 12:55:14 IST 2018.
> [Stage 7784:>(9 + 3) / 34][Stage 7786:(53 + 38) / 200][Stage 7789:(74 +
> 53) / 200]2018-05-27 12:57:32 WARN AsyncEventQueue:66 - Dropped
> com.codahale.metrics.Counter@1d8423d1 events from appStatus since Sun May
> 27 12:56:15 IST 2018.
> [Stage 7784:>(9 + 3) / 34][Stage 7786:(53 + 38) / 200][Stage 7789:(74 +
> 53) / 200]2018-05-27 12:58:32 WARN AsyncEventQueue:66 - Dropped
> com.codahale.metrics.Counter@1d8423d1 events from appStatus since Sun May
> 27 12:57:32 IST 2018.
> [Stage 7784:>(9 + 3) / 34][Stage 7786:(53 + 38) / 200][Stage 7789:(74 +
> 53) / 200]2018-05-27 12:59:33 WARN AsyncEventQueue:66 - Dropped
> com.codahale.metrics.Counter@1d8423d1 events from appStatus since Sun May
> 27 12:58:32 IST 2018.
> [Stage 7784:>(9 + 3) / 34][Stage 7786:(53 + 38) / 200][Stage 7789:(74 +
> 53) / 200]2018-05-27 13:00:34 WARN AsyncEventQueue:66 - Dropped
> com.codahale.metrics.Counter@1d8423d1 events from appStatus since Sun May
> 27 12:59:33 IST 2018.
> [Stage 7784:>(9 + 3) / 34][Stage 7786:(53 + 38) / 200][Stage 7789:(74 +
> 53) / 200]2018-05-27 13:01:35 WARN AsyncEventQueue:66 - Dropped
> com.codahale.metrics.Counter@1d8423d1 events from appStatus since Sun May
> 27 13:00:34 IST 2018.
> [Stage 7784:>(9 + 3) / 34][Stage 7786:(53 + 38) / 200][Stage 7789:(74 +
> 53) / 200]2018-05-27 13:02:36 WARN AsyncEventQueue:66 - Dropped
> com.codahale.metrics.Counter@1d8423d1 events from appStatus since Sun May
> 27 13:01:35 IST 2018.
>
Even though my job is not failing but why am I getting these?
Thanks,
Aakash.
Re: Spark AsyncEventQueue doubt
Posted by Yuanjian Li <xy...@gmail.com>.
Hi Askash
The event dropping problem also triggered by slow listener or large number of events or both, the easy and simple way is change the config of `spark.scheduler.listenerbus.eventqueue.capacity`, its default value is 10000. But if after change the queue capacity to a more lager value, it still happened, you should check more about listener.
> 在 2018年5月27日，15:45，Aakash Basu <aa...@gmail.com> 写道：
>
> Hi,
>
> I'm getting the below ERROR and WARN when running a little heavy calculation on a dataset -
>
> To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
> 2018-05-27 12:51:11 ERROR AsyncEventQueue:70 - Dropping event from queue appStatus. This likely means one of the listeners is too slow and cannot keep up with the rate at which tasks are being started by the scheduler.
> 2018-05-27 12:51:11 WARN AsyncEventQueue:66 - Dropped com.codahale.metrics.Counter@1d8423d1 events from appStatus since Thu Jan 01 05:30:00 IST 1970.
> [Stage 7784:>(9 + 3) / 34][Stage 7786:(53 + 38) / 200][Stage 7789:(74 + 53) / 200]2018-05-27 12:52:14 WARN AsyncEventQueue:66 - Dropped com.codahale.metrics.Counter@1d8423d1 events from appStatus since Sun May 27 12:51:11 IST 2018.
> [Stage 7784:>(9 + 3) / 34][Stage 7786:(53 + 38) / 200][Stage 7789:(74 + 53) / 200]2018-05-27 12:53:14 WARN AsyncEventQueue:66 - Dropped com.codahale.metrics.Counter@1d8423d1 events from appStatus since Sun May 27 12:52:14 IST 2018.
> 2018-05-27 12:54:14 WARN AsyncEventQueue:66 - Dropped com.codahale.metrics.Counter@1d8423d1 events from appStatus since Sun May 27 12:53:14 IST 2018.
> [Stage 7784:>(9 + 3) / 34][Stage 7786:(53 + 38) / 200][Stage 7789:(74 + 53) / 200]2018-05-27 12:55:14 WARN AsyncEventQueue:66 - Dropped com.codahale.metrics.Counter@1d8423d1 events from appStatus since Sun May 27 12:54:14 IST 2018.
> [Stage 7784:>(9 + 3) / 34][Stage 7786:(53 + 38) / 200][Stage 7789:(74 + 53) / 200]2018-05-27 12:56:15 WARN AsyncEventQueue:66 - Dropped com.codahale.metrics.Counter@1d8423d1 events from appStatus since Sun May 27 12:55:14 IST 2018.
> [Stage 7784:>(9 + 3) / 34][Stage 7786:(53 + 38) / 200][Stage 7789:(74 + 53) / 200]2018-05-27 12:57:32 WARN AsyncEventQueue:66 - Dropped com.codahale.metrics.Counter@1d8423d1 events from appStatus since Sun May 27 12:56:15 IST 2018.
> [Stage 7784:>(9 + 3) / 34][Stage 7786:(53 + 38) / 200][Stage 7789:(74 + 53) / 200]2018-05-27 12:58:32 WARN AsyncEventQueue:66 - Dropped com.codahale.metrics.Counter@1d8423d1 events from appStatus since Sun May 27 12:57:32 IST 2018.
> [Stage 7784:>(9 + 3) / 34][Stage 7786:(53 + 38) / 200][Stage 7789:(74 + 53) / 200]2018-05-27 12:59:33 WARN AsyncEventQueue:66 - Dropped com.codahale.metrics.Counter@1d8423d1 events from appStatus since Sun May 27 12:58:32 IST 2018.
> [Stage 7784:>(9 + 3) / 34][Stage 7786:(53 + 38) / 200][Stage 7789:(74 + 53) / 200]2018-05-27 13:00:34 WARN AsyncEventQueue:66 - Dropped com.codahale.metrics.Counter@1d8423d1 events from appStatus since Sun May 27 12:59:33 IST 2018.
> [Stage 7784:>(9 + 3) / 34][Stage 7786:(53 + 38) / 200][Stage 7789:(74 + 53) / 200]2018-05-27 13:01:35 WARN AsyncEventQueue:66 - Dropped com.codahale.metrics.Counter@1d8423d1 events from appStatus since Sun May 27 13:00:34 IST 2018.
> [Stage 7784:>(9 + 3) / 34][Stage 7786:(53 + 38) / 200][Stage 7789:(74 + 53) / 200]2018-05-27 13:02:36 WARN AsyncEventQueue:66 - Dropped com.codahale.metrics.Counter@1d8423d1 events from appStatus since Sun May 27 13:01:35 IST 2018.
>
> Even though my job is not failing but why am I getting these?
>
> Thanks,
> Aakash.