You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/03/11 03:14:53 UTC
[GitHub] [spark] liupc opened a new pull request #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
liupc opened a new pull request #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871
### What changes were proposed in this pull request?
Currently, spark will sort taskSets by jobId and stageId and then schedule them in order for FIFO schedulingMode. In OLAP senerios, especially under high concurrency, the taskSets are always from different sql queries and several jobs can be submitted for execution at one time for one query, especailly for adaptive execution. But now we order those taskSets without considering the execution group, which may causes the query being delayed.
So I propose to consider the sql execution id when scheduling jobs.
### Why are the changes needed?
Improvements
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
existing UT & added UT
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-602139486
Hi, @liupc .
- Currently, `Core` module's `FIFO` concept doesn't mean SQL level execution. The following is not Apache Spark's contract.
> And users expect the FIFO behave like this.
- Also, this is a trade-off. And, in general, this hurts the global throughput (as @jiangxb1987 mentioned already). Apache Spark cannot accept that kind of general performance degradation for those rare use cases. Please note that not every users are SQL users.
- Lastly, I'm -1 for the current architectural design which makes a dependency from `core` module's `TaskSetManager` to the external `sql` module.
Given that, this PR will not considered as a mergeable PR. If you want to proceed, please split the scopes. First, you may need to focus on adding a new option to respect `job` level priority in the `core` module first. The configuration should be `false` by default. After the suggestion is accepted into `master` branch, you may need to make a second PR to add another option for the whole SQL Query optimization.
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-599795411
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119893/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-600302083
Ping, @liupc . If there is no other reason, shall we close this PR according to the review comments?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liupc commented on issue #27871: [SPARK-31105][CORE]Respect
sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
liupc commented on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-601598502
> For small queries, usually they won't hit this problem. For big queries, the query latency shouldn't matter too much?
>
> @liupc have you tried this in real-world workloads?
Yes, we tried in real workloads, it does better especially there are lots of taskSets to be scheduled for one round scheduling. This is obvious for adaptive execution. Also, I think this is what FIFO should do.
usually queries may mapping to several jobs, if several jobs being delayed due to this reason, the total delay is obvious.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liupc edited a comment on issue #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
liupc edited a comment on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-601598502
> For small queries, usually they won't hit this problem. For big queries, the query latency shouldn't matter too much?
>
> @liupc have you tried this in real-world workloads?
Yes, we tried in real workloads, it does better especially there are lots of taskSets to be scheduled for one round scheduling. This is obvious for adaptive execution. Also, I think this is what FIFO should do.
usually queries may mapping to several jobs, if several jobs being delayed due to this reason, the total delay is obvious. Suppose each job duration would be 2 min, then if there are 10 jobs in front of the job and the cores is fully used. then due to this reason, it wait 20min to be scheduled.What's worse, in adaptive exeuction, when next batch of jobs being submitted, it may met this issue again, that may greatly harm the query duration.
Also, users will see lots of jobs running for later comming queries in SparkUI, that's confusing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-599938120
+1 for @jiangxb1987 's comments. BTW, I guessed that that's the main reason @liupc tried this at `FIFO` scheduler.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-597423397
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-599740322
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-599954941
For small queries, usually they won't hit this problem. For big queries, the query latency shouldn't matter too much?
@liupc have you tried this in real-world workloads?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-599739794
**[Test build #119893 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119893/testReport)** for PR 27871 at commit [`001b234`](https://github.com/apache/spark/commit/001b2340859c02be845598869bacc7e96ac34d2f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-599738592
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jiangxb1987 commented on issue #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
jiangxb1987 commented on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-599909004
IIUC Spark didn't optimize the workload toward the direction that the query delayed time should be minimized. Actually, I think scheduling Stages from the same sql execution would lead to a few nodes with the shuffle output files become very hot, thus lead to worse performance for the whole cluster compared to current approach.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liupc edited a comment on issue #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
liupc edited a comment on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-602373911
Thanks @dongjoon-hyun , let's spill the scopes, and add an option to respect `jobGroup` level priority in the `core` module.
And I think even in current approach, the congestion issue is serious, so this PR is not about to solve it, but I proposed another PR for this: https://github.com/apache/spark/pull/27862
I really think this is helpful for OLAP senarios, and we test this in real workloads in xiaomi.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-597423687
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#discussion_r393307959
##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala
##########
@@ -111,8 +111,11 @@ private[spark] class TaskSetManager(
val weight = 1
val minShare = 0
- var priority = taskSet.priority
- var stageId = taskSet.stageId
+ val priority = if (taskSet.properties != null) {
+ taskSet.properties.getProperty("spark.sql.execution.id", "0").toLong
Review comment:
Although we has a similar one inside `AppStatusListener` already, this means another implicit dependency to `sql` module.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#discussion_r393308617
##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala
##########
@@ -111,8 +111,11 @@ private[spark] class TaskSetManager(
val weight = 1
val minShare = 0
- var priority = taskSet.priority
- var stageId = taskSet.stageId
+ val priority = if (taskSet.properties != null) {
+ taskSet.properties.getProperty("spark.sql.execution.id", "0").toLong
Review comment:
BTW, you may want to define `private val SQL_EXECUTION_ID_KEY = "spark.sql.execution.id"` like `AppStatusListener`. Or, if we need this, we may need to define this at the more higher common place.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liupc edited a comment on issue #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
liupc edited a comment on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-601598502
> For small queries, usually they won't hit this problem. For big queries, the query latency shouldn't matter too much?
>
> @liupc have you tried this in real-world workloads?
Yes, we tried in real workloads, it does better especially there are lots of taskSets to be scheduled for one round scheduling. This is obvious for adaptive execution. Also, I think this is what FIFO should do.
usually queries may mapping to several jobs, if several jobs being delayed due to this reason, the total delay is obvious. Suppose each job duration would be 2 min, then if there are 10 jobs in front of the job and the cores is fully used. then due to this reason, it wait 20min to be scheduled.What's worse, in adaptive exeuction, when next batch of jobs being submitted, it may met this issue again, that may greatly harm the query duration. And each query may have the same issue and thus slow down them all.
Also, users will see lots of jobs running for later comming queries in SparkUI, that's confusing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jiangxb1987 commented on issue #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
jiangxb1987 commented on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-599909770
Also, please consider the case you submit a sql query that requires so many slots that it would block all the other queries from being executed. In the current approach, other small queries still get a chance to execute between two huge stages, after your change, every small queries need to wait until the first big query finishes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-599740331
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24619/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#discussion_r393310576
##########
File path: core/src/test/scala/org/apache/spark/scheduler/PoolSuite.scala
##########
@@ -344,4 +344,36 @@ class PoolSuite extends SparkFunSuite with LocalSparkContext {
assert(selectedPool.weight === expectedInitWeight)
assert(selectedPool.schedulingMode === expectedSchedulingMode)
}
+
+ test("SPARK-31105: FIFO Scheduler should respect execution id") {
+ val conf = new SparkConf()
+ sc = new SparkContext(LOCAL, APP_NAME, conf)
+ val taskScheduler = new TaskSchedulerImpl(sc)
+
+ val rootPool = new Pool("", FIFO, 0, 0)
+ val schedulableBuilder = new FairSchedulableBuilder(rootPool, conf)
Review comment:
`FairSchedulableBuilder` -> `FIFOSchedulableBuilder`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liupc edited a comment on issue #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
liupc edited a comment on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-602373911
Thanks @dongjoon-hyun , let's spill the scopes, and add an option to respect `jobGroup` level priority in the `core` module.
I really think this is helpful for OLAP senarios, and that's what we do in xiaomi.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-602816245
Thank you, @liupc . I hope Apache Spark can improve your use cases in your enviornment in any way. Please move forward. I'll close this PR (AS-IS). You can reopen this later after the `core` module is ready independently.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-599795411
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119893/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-597423687
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-599740331
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24619/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liupc commented on issue #27871: [SPARK-31105][CORE]Respect
sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
liupc commented on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-602373911
Thanks @dongjoon-hyun , let's spill the scopes, and add an option to respect `jobGroup` level priority in the `core` module.
I really think this is helpful for OLAP senarios.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-599795121
**[Test build #119893 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119893/testReport)** for PR 27871 at commit [`001b234`](https://github.com/apache/spark/commit/001b2340859c02be845598869bacc7e96ac34d2f).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-599759116
How do you think about this, @gatorsmile and @cloud-fan ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-599739794
**[Test build #119893 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119893/testReport)** for PR 27871 at commit [`001b234`](https://github.com/apache/spark/commit/001b2340859c02be845598869bacc7e96ac34d2f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-599795406
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liupc commented on issue #27871: [SPARK-31105][CORE]Respect
sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
liupc commented on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-601599878
`
> IIUC Spark didn't optimize the workload toward the direction that the query delayed time should be minimized. Actually, I think scheduling Stages from the same sql execution would lead to a few nodes with the shuffle output files become very hot, thus lead to worse performance for the whole cluster compared to current approach.
In real clusters, resources is more important than locality. And users expect the FIFO behave like this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-599740322
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-599795406
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liupc edited a comment on issue #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
liupc edited a comment on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-601598502
> For small queries, usually they won't hit this problem. For big queries, the query latency shouldn't matter too much?
>
> @liupc have you tried this in real-world workloads?
Yes, we tried in real workloads, it does better especially there are lots of taskSets to be scheduled for one round scheduling. This is obvious for adaptive execution. Also, I think this is what FIFO should do.
usually queries may mapping to several jobs, if several jobs being delayed due to this reason, the total delay is obvious. Suppose each job duration would be 2 min, then if there are 10 jobs in front of the job and the cores is fully used. then due to this reason, it wait 20min to be scheduled.
Also, users will see lots of jobs running for later comming queries in SparkUI, that's confusing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27871:
[SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
URL: https://github.com/apache/spark/pull/27871#issuecomment-597423397
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org