You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Shing Hing Man <ma...@yahoo.com.INVALID> on 2018/10/19 23:41:36 UTC
Spark 2.3.2 : No of active tasks vastly exceeds total no of
executor cores
Hi, I have just upgraded my application to Spark 2.3.2 from 2.2.1. When I run my Spark application in Yarn, in the executor tab of Spark UI, I see there are 1499 active tasks.There is only 145 cores in my executors. I have not changed any of spark.ui.* parameters.
In Spark 2.2.1, the number of active tasks never exceeds 145 cores, the total no of cpu cores of all the executors.
Also my application takes 4 times longer to run with Spark 2.3.2 than with Spark 2.2.1.
I wonder if my application is slow down because of too many active tasks.
Thanks in advance for any assistance !
Shing
Re: Spark 2.3.2 : No of active tasks vastly exceeds total no of
executor cores
Posted by Shing Hing Man <ma...@yahoo.com.INVALID>.
I have increased spark.scheduler.listenerbus.eventqueue.capacity, and ran my application (in Yarn client mode) as before. I no longer get "Dropped events". But the driver ran out of memory. The Spark UI gradually became unreponsive. I noticed from the Spark UI
that tens of thousands of jobs were kept. In my application. many Spark jobs are submitted concurrently. I think I am seeing the problem mentioned in the following recent post.
Apache Spark User List - [Spark UI] Spark 2.3.1 UI no longer respects spark.ui.retainedJobs
|
|
|
| | |
|
|
|
| |
Apache Spark User List - [Spark UI] Spark 2.3.1 UI no longer respects sp...
[Spark UI] Spark 2.3.1 UI no longer respects spark.ui.retainedJobs. I recently upgraded to spark 2.3.1 I have ha...
|
|
|
Shing
On Monday, 22 October 2018, 19:56:32 GMT+1, Shing Hing Man <ma...@yahoo.com> wrote:
In my log, I have found
mylog.2:2018-10-19 20:00:50,455 WARN [dag-scheduler-event-loop] (Logging.scala:66) - Dropped 3498 events from appStatus since Fri Oct 19 19:25:05 UTC 2018.mylog.2:2018-10-19 20:02:07,053 WARN [dispatcher-event-loop-1] (Logging.scala:66) - Dropped 123385 events from appStatus since Fri Oct 19 20:00:50 UTC 2018.mylog.3:2018-10-19 19:23:42,922 ERROR [dispatcher-event-loop-3] (Logging.scala:70) - Dropping event from queue appStatus. This likely means one of the listeners is too slow and cannot keep up with the rate at which tasks are being started by the scheduler.mylog.3:2018-10-19 19:23:42,928 WARN [dag-scheduler-event-loop] (Logging.scala:66) - Dropped 2 events from appStatus since Thu Jan 01 00:00:00 UTC 1970.mylog.3:2018-10-19 19:25:05,822 WARN [dag-scheduler-event-loop] (Logging.scala:66) - Dropped 12190 events from appStatus since Fri Oct 19 19:23:42 UTC 2018.
I will try increasing spark.scheduler.listenerbus.eventqueue.capacity ,
Shing
On Monday, 22 October 2018, 01:46:11 BST, Mark Hamstra <ma...@clearstorydata.com> wrote:
Look for these log messages:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala#L154
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala#L172
On Fri, Oct 19, 2018 at 4:42 PM Shing Hing Man <ma...@yahoo.com.invalid> wrote:
Hi, I have just upgraded my application to Spark 2.3.2 from 2.2.1. When I run my Spark application in Yarn, in the executor tab of Spark UI, I see there are 1499 active tasks.There is only 145 cores in my executors. I have not changed any of spark.ui.* parameters.
In Spark 2.2.1, the number of active tasks never exceeds 145 cores, the total no of cpu cores of all the executors.
Also my application takes 4 times longer to run with Spark 2.3.2 than with Spark 2.2.1.
I wonder if my application is slow down because of too many active tasks.
Thanks in advance for any assistance !
Shing
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Spark 2.3.2 : No of active tasks vastly exceeds total no of
executor cores
Posted by Shing Hing Man <ma...@yahoo.com.INVALID>.
In my log, I have found
mylog.2:2018-10-19 20:00:50,455 WARN [dag-scheduler-event-loop] (Logging.scala:66) - Dropped 3498 events from appStatus since Fri Oct 19 19:25:05 UTC 2018.mylog.2:2018-10-19 20:02:07,053 WARN [dispatcher-event-loop-1] (Logging.scala:66) - Dropped 123385 events from appStatus since Fri Oct 19 20:00:50 UTC 2018.mylog.3:2018-10-19 19:23:42,922 ERROR [dispatcher-event-loop-3] (Logging.scala:70) - Dropping event from queue appStatus. This likely means one of the listeners is too slow and cannot keep up with the rate at which tasks are being started by the scheduler.mylog.3:2018-10-19 19:23:42,928 WARN [dag-scheduler-event-loop] (Logging.scala:66) - Dropped 2 events from appStatus since Thu Jan 01 00:00:00 UTC 1970.mylog.3:2018-10-19 19:25:05,822 WARN [dag-scheduler-event-loop] (Logging.scala:66) - Dropped 12190 events from appStatus since Fri Oct 19 19:23:42 UTC 2018.
I will try increasing spark.scheduler.listenerbus.eventqueue.capacity ,
Shing
On Monday, 22 October 2018, 01:46:11 BST, Mark Hamstra <ma...@clearstorydata.com> wrote:
Look for these log messages:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala#L154
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala#L172
On Fri, Oct 19, 2018 at 4:42 PM Shing Hing Man <ma...@yahoo.com.invalid> wrote:
Hi, I have just upgraded my application to Spark 2.3.2 from 2.2.1. When I run my Spark application in Yarn, in the executor tab of Spark UI, I see there are 1499 active tasks.There is only 145 cores in my executors. I have not changed any of spark.ui.* parameters.
In Spark 2.2.1, the number of active tasks never exceeds 145 cores, the total no of cpu cores of all the executors.
Also my application takes 4 times longer to run with Spark 2.3.2 than with Spark 2.2.1.
I wonder if my application is slow down because of too many active tasks.
Thanks in advance for any assistance !
Shing
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Spark 2.3.2 : No of active tasks vastly exceeds total no of
executor cores
Posted by Mark Hamstra <ma...@clearstorydata.com>.
Look for these log messages:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala#L154
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala#L172
On Fri, Oct 19, 2018 at 4:42 PM Shing Hing Man <ma...@yahoo.com.invalid>
wrote:
> Hi,
> I have just upgraded my application to Spark 2.3.2 from 2.2.1. When I
> run my Spark application in Yarn, in the executor tab of Spark UI, I see
> there are 1499 active tasks.
> There is only 145 cores in my executors. I have not changed any of
> spark.ui.* parameters.
>
> In Spark 2.2.1, the number of active tasks never exceeds 145 cores, the
> total no of cpu cores of all the executors.
>
> Also my application takes 4 times longer to run with Spark 2.3.2 than with
> Spark 2.2.1.
>
> I wonder if my application is slow down because of too many active tasks.
>
>
> [image: Inline image]
>
>
> Thanks in advance for any assistance !
> Shing
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org