You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dongjoon-hyun (via GitHub)" <gi...@apache.org> on 2026/08/20 15:29:28 UTC
[PR] [SPARK-58913][CORE] Add `TaskSchedulerImpl.hasPipelinedTaskSets` [spark]
dongjoon-hyun opened a new pull request, #58171:
URL: https://github.com/apache/spark/pull/58171
### What changes were proposed in this pull request?
This PR adds `TaskSchedulerImpl.hasPipelinedTaskSets`, a query for whether any live task set belongs to a pipelined group:
```scala
private[spark] def hasPipelinedTaskSets: Boolean = synchronized {
taskSetsByStageIdAndAttempt.values.exists(_.values.exists { tsm =>
!tsm.isZombie && tsm.taskSet.isPipelined
})
}
```
It sits next to `outstandingTasksForOtherWorkInProfile`, the other pipelined-group helper over the same map, and follows its conventions: taken under the existing lock, and skipping zombie (superseded) attempts.
This is a spin-off of https://github.com/apache/spark/pull/58054, which introduces `SparkContext.holdExecutors()` and consumes this method.
### Why are the changes needed?
A pipelined group's shuffle data is transient and lives only on the executors running the group, so a caller about to disturb those executors -- graceful decommission in #58054 -- needs to know that such a group is running. No existing API answers that: `TaskSet.isPipelined` is per task set, and `taskSetsByStageIdAndAttempt` is private to `TaskSchedulerImpl`.
The check is best-effort by design. It reports what the *task* scheduler currently holds, which is narrower than "a pipelined job is active": it is false before the group's first task set is submitted, and false again once the last member's `TaskSetManager` has gone zombie but the `DAGScheduler` has not yet processed the final completion event.
### Does this PR introduce _any_ user-facing change?
No. `hasPipelinedTaskSets` is `private[spark]` and has no caller in this PR.
### How was this patch tested?
Three new unit tests in `TaskSchedulerImplSuite` cover a live pipelined task set, the `!isZombie` filter in both directions, and the tail of the best-effort window.
```
build/sbt 'core/testOnly *TaskSchedulerImplSuite -- -z SPARK-58913'
```
```
[info] Tests: succeeded 3, failed 0, canceled 0, ignored 0, pending 0
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58913][CORE] Add `TaskSchedulerImpl.hasPipelinedTaskSets` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #58171:
URL: https://github.com/apache/spark/pull/58171#issuecomment-5360100394
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/020da8cc090e40f5700481d4b82cff955c57fd63
- merged into branch-4.x https://github.com/apache/spark/commit/977c8f19db9d40b500911fdfb3e4eaa86a887439
- merged into branch-4.3 https://github.com/apache/spark/commit/e87de79713887b801ed675221a50d3366adc0655
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58913][CORE] Add `TaskSchedulerImpl.hasPipelinedTaskSets` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #58171:
URL: https://github.com/apache/spark/pull/58171#issuecomment-5358596393
Thank you so much, @LuciferYang !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58913][CORE] Add `TaskSchedulerImpl.hasPipelinedTaskSets` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #58171:
URL: https://github.com/apache/spark/pull/58171#issuecomment-5358156584
Could you review this PR, @LuciferYang ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58913][CORE] Add `TaskSchedulerImpl.hasPipelinedTaskSets` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #58171: [SPARK-58913][CORE] Add `TaskSchedulerImpl.hasPipelinedTaskSets`
URL: https://github.com/apache/spark/pull/58171
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org