You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "dingbei (Jira)" <ji...@apache.org> on 2020/11/11 07:06:00 UTC
[jira] [Updated] (SPARK-33418) TaskSchedulerImpl: Check pending
tasks in advance when resource offers
[ https://issues.apache.org/jira/browse/SPARK-33418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dingbei updated SPARK-33418:
----------------------------
Description:
It begins with the needs to start a lot of spark streaming receivers . The launch time gets super long when it comes to more than 300 receivers. I will show Tests data I did and how did I improve this. There are two cores in every executors.(one for receiver and the other one to process bacth jobs),and there will be a batch of data every 10s.
There is two main metrics i will mention below.
receiver launch time :From the first receiver started to the last one.(observed through spark web UI)
batch data processing time: the time it takes to process a batch of data.(observed through spark web UI streaming)
At first, we set the number of executors to 200 which means to start 200 receivers and everything goes well. launch time is around 50s.processing time for every batch is around 10s.
Then we set the number of executors to 500 which means to start 500 receivers
> TaskSchedulerImpl: Check pending tasks in advance when resource offers
> ----------------------------------------------------------------------
>
> Key: SPARK-33418
> URL: https://issues.apache.org/jira/browse/SPARK-33418
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.0.1
> Reporter: dingbei
> Priority: Major
>
> It begins with the needs to start a lot of spark streaming receivers . The launch time gets super long when it comes to more than 300 receivers. I will show Tests data I did and how did I improve this. There are two cores in every executors.(one for receiver and the other one to process bacth jobs),and there will be a batch of data every 10s.
>
> There is two main metrics i will mention below.
> receiver launch time :From the first receiver started to the last one.(observed through spark web UI)
> batch data processing time: the time it takes to process a batch of data.(observed through spark web UI streaming)
>
> At first, we set the number of executors to 200 which means to start 200 receivers and everything goes well. launch time is around 50s.processing time for every batch is around 10s.
> Then we set the number of executors to 500 which means to start 500 receivers
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org