You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by liyichao <gi...@git.apache.org> on 2017/05/24 17:55:57 UTC
[GitHub] spark pull request #18093: [SPARK-20774][SQL] Cancel all jobs when QueryExec...
GitHub user liyichao opened a pull request:
https://github.com/apache/spark/pull/18093
[SPARK-20774][SQL] Cancel all jobs when QueryExection throws.
see https://issues.apache.org/jira/browse/SPARK-20774?filter=12340455
## What changes were proposed in this pull request?
When the sql query failed, all related jobs are canceled to reap resources.
## How was this patch tested?
no.
cc @zsxwing
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/liyichao/spark SPARK-20774
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18093.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18093
----
commit 32e5c26e6861a4f14bcea552e68854a8b452ada7
Author: Li Yichao <ly...@zhihu.com>
Date: 2017-05-22T08:59:46Z
Cancel all jobs when QueryExection throws.
see https://issues.apache.org/jira/browse/SPARK-20774?filter=12340455
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18093: [WIP][SPARK-20774][SQL] Cancel all jobs when QueryExecti...
Posted by liyichao <gi...@git.apache.org>.
Github user liyichao commented on the issue:
https://github.com/apache/spark/pull/18093
Sorry about that, I will test it when I have time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18093: [SPARK-20774][SQL] Cancel all jobs when QueryExection th...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18093
I think a patch should at least be manually tested @liyichao.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18093: [WIP][SPARK-20774][SQL] Cancel all jobs when Quer...
Posted by markhamstra <gi...@git.apache.org>.
Github user markhamstra commented on a diff in the pull request:
https://github.com/apache/spark/pull/18093#discussion_r129382985
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala ---
@@ -89,8 +91,22 @@ class QueryExecution(val sparkSession: SparkSession, val logical: LogicalPlan) {
lazy val executedPlan: SparkPlan = prepareForExecution(sparkPlan)
/** Internal version of the RDD. Avoids copies and has no schema */
- lazy val toRdd: RDD[InternalRow] = executedPlan.execute()
+ lazy val toRdd: RDD[InternalRow] = executePlan()
+ private def executePlan(): RDD[InternalRow] = {
+ sparkSession.sparkContext.setJobGroup(runId.toString, getDescriptionString,
--- End diff --
Wait... aren't you clobbering any JobGroup that is already set?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18093: [WIP][SPARK-20774][SQL] Cancel all jobs when Quer...
Posted by liyichao <gi...@git.apache.org>.
Github user liyichao closed the pull request at:
https://github.com/apache/spark/pull/18093
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18093: [SPARK-20774][SQL] Cancel all jobs when QueryExection th...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18093
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18093: [WIP][SPARK-20774][SQL] Cancel all jobs when QueryExecti...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18093
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org