You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Bowen Song <bo...@kyligence.io> on 2022/05/18 03:09:13 UTC
Re: A scene with unstable Spark performance
Hi,
Spark dynamic resource allocation cannot solve my problem, because the resources of the production environment are limited. I expect that under this premise, by reserving resources to ensure that job tasks of different groups can be scheduled in time.
Thank you,
Bowen Song
________________________________
From: Qian SUN <qi...@gmail.com>
Sent: Wednesday, May 18, 2022 9:32
To: Bowen Song <bo...@kyligence.io>
Cc: user.spark <us...@spark.apache.org>
Subject: Re: A scene with unstable Spark performance
Hi. I think you need Spark dynamic resource allocation. Please refer to https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation.
And If you use Spark SQL, AQE maybe help. https://spark.apache.org/docs/latest/sql-performance-tuning.html#adaptive-query-execution
Bowen Song <bo...@kyligence.io>> 于2022年5月17日周二 22:33写道：
Hi all,
I find Spark performance is unstable in this scene: we divided the jobs into two groups according to the job completion time. One group of jobs had an execution time of less than 10s, and the other group of jobs had an execution time from 10s to 300s. The reason for the difference is that the latter will scan more files, that is, the number of tasks will be larger. When the two groups of jobs were submitted to Spark for execution, I found that due to resource competition, the existence of the slower jobs made the original faster job take longer to return the result, which manifested as unstable Spark performance. The problem I want to solve is: Can we reserve certain resources for each of the two groups, so that the fast jobs can be scheduled in time, and the slow jobs will not be starved to death because the resources are completely allocated to the fast jobs.
In this context, I need to group spark jobs, and the tasks from different groups of jobs can be scheduled using group reserved resources. At the beginning of each round of scheduling, tasks in this group will be scheduled first, only when there are no tasks in this group to schedule, its resources can be allocated to other groups to avoid idling of resources.
For the consideration of resource utilization and the overhead of managing multiple clusters, I hope that the jobs can share the spark cluster, rather than creating private clusters for the groups.
I've read the code for the Spark Fair Scheduler, and the implementation doesn't seem to meet the need to reserve resources for different groups of job.
Is there a workaround that can solve this problem through Spark Fair Scheduler? If it can't be solved, would you consider adding a mechanism like capacity scheduling.
Thank you,
Bowen Song
--
Best!
Qian SUN
Re: A scene with unstable Spark performance
Posted by Chang Chen <ba...@gmail.com>.
This is a case where resources are fixed in the same SparkContext, but sqls
have different priorities.
Some SQLs are only allowed to be executed if there are spare resources,
once the high priority sql comes in, those sqls taskset either are killed
or stalled.
If we set a high priority pool's minShare to a relatively higher value,
e.g. 50% or 60% of total cores, does it make sense?
Sungwoo Park <gl...@gmail.com> 于2022年5月18日周三 13:28写道：
> The problem you describe is the motivation for developing Spark on MR3.
> From the blog article (
> https://www.datamonad.com/post/2021-08-18-spark-mr3/):
>
> *The main motivation for developing Spark on MR3 is to allow multiple
> Spark applications to share compute resources such as Yarn containers or
> Kubernetes Pods.*
>
> The problem is due to an architectural limitation of Spark, and I guess
> fixing the problem would require a heavy rewrite of Spark core. When we
> developed Spark on MR3, we were not aware of any attempt being made
> elsewhere (in academia and industry) to address this limitation.
>
> A potential workaround might be to implement a custom Spark application
> that manages the submission of two groups of Spark jobs and controls their
> execution (similarly to Spark Thrift Server). Not sure if this approach
> would fix your problem, though.
>
> If you are interested, see the webpage of Spark on MR3:
> https://mr3docs.datamonad.com/docs/spark/
>
> We have released Spark 3.0.1 on MR3, and Spark 3.2.1 on MR3 is under
> development. For Spark 3.0.1 on MR3, no change is made to Spark and MR3 is
> used as an add-on. The main application of MR3 is Hive on MR3, but Spark on
> MR3 is equally ready for production.
>
> Thank you,
>
> --- Sungwoo
>
>>
Re: A scene with unstable Spark performance
Posted by Chang Chen <ba...@gmail.com>.
This is a case where resources are fixed in the same SparkContext, but sqls
have different priorities.
Some SQLs are only allowed to be executed if there are spare resources,
once the high priority sql comes in, those sqls taskset either are killed
or stalled.
If we set a high priority pool's minShare to a relatively higher value,
e.g. 50% or 60% of total cores, does it make sense?
Sungwoo Park <gl...@gmail.com> 于2022年5月18日周三 13:28写道：
> The problem you describe is the motivation for developing Spark on MR3.
> From the blog article (
> https://www.datamonad.com/post/2021-08-18-spark-mr3/):
>
> *The main motivation for developing Spark on MR3 is to allow multiple
> Spark applications to share compute resources such as Yarn containers or
> Kubernetes Pods.*
>
> The problem is due to an architectural limitation of Spark, and I guess
> fixing the problem would require a heavy rewrite of Spark core. When we
> developed Spark on MR3, we were not aware of any attempt being made
> elsewhere (in academia and industry) to address this limitation.
>
> A potential workaround might be to implement a custom Spark application
> that manages the submission of two groups of Spark jobs and controls their
> execution (similarly to Spark Thrift Server). Not sure if this approach
> would fix your problem, though.
>
> If you are interested, see the webpage of Spark on MR3:
> https://mr3docs.datamonad.com/docs/spark/
>
> We have released Spark 3.0.1 on MR3, and Spark 3.2.1 on MR3 is under
> development. For Spark 3.0.1 on MR3, no change is made to Spark and MR3 is
> used as an add-on. The main application of MR3 is Hive on MR3, but Spark on
> MR3 is equally ready for production.
>
> Thank you,
>
> --- Sungwoo
>
>>
Re: A scene with unstable Spark performance
Posted by Sungwoo Park <gl...@gmail.com>.
The problem you describe is the motivation for developing Spark on MR3.
From the blog article (https://www.datamonad.com/post/2021-08-18-spark-mr3/
):
*The main motivation for developing Spark on MR3 is to allow multiple Spark
applications to share compute resources such as Yarn containers or
Kubernetes Pods.*
The problem is due to an architectural limitation of Spark, and I guess
fixing the problem would require a heavy rewrite of Spark core. When we
developed Spark on MR3, we were not aware of any attempt being made
elsewhere (in academia and industry) to address this limitation.
A potential workaround might be to implement a custom Spark application
that manages the submission of two groups of Spark jobs and controls their
execution (similarly to Spark Thrift Server). Not sure if this approach
would fix your problem, though.
If you are interested, see the webpage of Spark on MR3:
https://mr3docs.datamonad.com/docs/spark/
We have released Spark 3.0.1 on MR3, and Spark 3.2.1 on MR3 is under
development. For Spark 3.0.1 on MR3, no change is made to Spark and MR3 is
used as an add-on. The main application of MR3 is Hive on MR3, but Spark on
MR3 is equally ready for production.
Thank you,
--- Sungwoo
>