You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/03/10 10:31:15 UTC
[GitHub] [spark] liupc opened a new pull request #27862:
[SPARK-31107][CORE]Extend FairScheduler to support pool level resource
isolation
liupc opened a new pull request #27862: [SPARK-31107][CORE]Extend FairScheduler to support pool level resource isolation
URL: https://github.com/apache/spark/pull/27862
### What changes were proposed in this pull request?
Currently, spark only provided two types of scheduler: FIFO & FAIR, but in sql high-concurrency scenarios, a few of drawbacks are exposed.
FIFO: it can easily causing congestion when large sql query occupies all the resources
FAIR: the taskSets of one pool may occupies all the resource due to there are no hard limit on the maximum usage for each pool. this case may be frequently met under high workloads.
So we propose to add a maxShare argument for FairScheduler to control the maximum running tasks for each pool.
### Why are the changes needed?
Improvements
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
existing UT
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27862:
[SPARK-31107][CORE]Extend FairScheduler to support pool level resource
isolation
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27862: [SPARK-31107][CORE]Extend FairScheduler to support pool level resource isolation
URL: https://github.com/apache/spark/pull/27862#issuecomment-597014725
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27862:
[SPARK-31107][CORE]Extend FairScheduler to support pool level resource
isolation
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27862: [SPARK-31107][CORE]Extend FairScheduler to support pool level resource isolation
URL: https://github.com/apache/spark/pull/27862#issuecomment-597014725
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27862:
[SPARK-31107][CORE]Extend FairScheduler to support pool level resource
isolation
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27862: [SPARK-31107][CORE]Extend FairScheduler to support pool level resource isolation
URL: https://github.com/apache/spark/pull/27862#issuecomment-597015242
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org