You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by zsxwing <gi...@git.apache.org> on 2017/04/25 18:23:07 UTC
[GitHub] spark pull request #17761: [SPARK-20461][Core][SS]Use UninterruptibleThread ...
GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/17761
[SPARK-20461][Core][SS]Use UninterruptibleThread for Executor and fix the potential hang in CachedKafkaConsumer
## What changes were proposed in this pull request?
This PR changes Executor's threads to `UninterruptibleThread` so that we can use `runUninterruptibly` in `CachedKafkaConsumer`. However, this is just best effort to avoid hanging forever. If the user uses`CachedKafkaConsumer` in another thread (e.g., create a new thread or Future), the potential hang may still happen.
## How was this patch tested?
The new added test.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zsxwing/spark int
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17761.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17761
----
commit 3d7fc4d86107f1171321946adf58dbd9dbb1f9a1
Author: Shixiong Zhu <sh...@databricks.com>
Date: 2017-04-25T18:16:46Z
Use UninterruptibleThread for Executor and fix the potential hang in CachedKafkaConsumer
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17761: [SPARK-20461][Core][SS]Use UninterruptibleThread for Exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17761
**[Test build #76144 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76144/testReport)** for PR 17761 at commit [`3d7fc4d`](https://github.com/apache/spark/commit/3d7fc4d86107f1171321946adf58dbd9dbb1f9a1).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17761: [SPARK-20461][Core][SS]Use UninterruptibleThread for Exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17761
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76145/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17761: [SPARK-20461][Core][SS]Use UninterruptibleThread for Exe...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/17761
@zsxwing Got it, thanks for clarifying.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17761: [SPARK-20461][Core][SS]Use UninterruptibleThread for Exe...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17761
@mridulm this only affect codes calling `runUninterruptibly` which is not a public API, so won't break any existing codes. The worst case of this PR is some task needs to wait until network timeout. However, if removing `runUninterruptibly` added in this PR, it's pretty easy to be stuck in an endless loop of Kafka client when interrupting it. That's wasting resources and unacceptable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17761: [SPARK-20461][Core][SS]Use UninterruptibleThread for Exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17761
**[Test build #76145 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76145/testReport)** for PR 17761 at commit [`bde6120`](https://github.com/apache/spark/commit/bde6120320bb8a4abc68597b0c021d9284998f93).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17761: [SPARK-20461][Core][SS]Use UninterruptibleThread ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/17761
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17761: [SPARK-20461][Core][SS]Use UninterruptibleThread ...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/17761#discussion_r113273935
--- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/CachedKafkaConsumer.scala ---
@@ -62,11 +63,20 @@ private[kafka010] case class CachedKafkaConsumer private(
case class AvailableOffsetRange(earliest: Long, latest: Long)
+ private def runUninterruptiblyIfPossiable[T](body: => T): T = Thread.currentThread match {
--- End diff --
Possiable -> Possible
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17761: [SPARK-20461][Core][SS]Use UninterruptibleThread for Exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17761
**[Test build #76144 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76144/testReport)** for PR 17761 at commit [`3d7fc4d`](https://github.com/apache/spark/commit/3d7fc4d86107f1171321946adf58dbd9dbb1f9a1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17761: [SPARK-20461][Core][SS]Use UninterruptibleThread for Exe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17761
**[Test build #76145 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76145/testReport)** for PR 17761 at commit [`bde6120`](https://github.com/apache/spark/commit/bde6120320bb8a4abc68597b0c021d9284998f93).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17761: [SPARK-20461][Core][SS]Use UninterruptibleThread ...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17761#discussion_r113273103
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -84,7 +86,20 @@ private[spark] class Executor(
}
// Start worker thread pool
- private val threadPool = ThreadUtils.newDaemonCachedThreadPool("Executor task launch worker")
+ private val threadPool = {
+ val threadFactory = new ThreadFactoryBuilder()
+ .setDaemon(true)
+ .setNameFormat("Executor task launch worker-%d")
+ .setThreadFactory(new ThreadFactory {
+ override def newThread(r: Runnable): Thread =
+ // Use UninterruptibleThread to run tasks so that we can allow running codes without being
+ // interrupted by `Thread.interrupt()`. Some issues, such as KAFKA-1894, HADOOP-10622,
+ // will hang forever if some methods are interrupted.
+ new UninterruptibleThread(r, "unused") // thread name will be set by ThreadFactoryBuilder
--- End diff --
Mostly of the codes are copied from `ThreadUtils`. This one is the only difference that matters.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17761: [SPARK-20461][Core][SS]Use UninterruptibleThread for Exe...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/17761
Task interruption is part of our API iirc - kill with interrupt = true.
Doing this will break that contract - did I miss something here ?
Also, IIRC @joshrosen had done a bunch of work regarding task reaper, etc - which should get affected by this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17761: [SPARK-20461][Core][SS]Use UninterruptibleThread for Exe...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/17761
LGTM. Merging this to master and 2.2
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17761: [SPARK-20461][Core][SS]Use UninterruptibleThread ...
Posted by kunalkhamar <gi...@git.apache.org>.
Github user kunalkhamar commented on a diff in the pull request:
https://github.com/apache/spark/pull/17761#discussion_r113274024
--- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/CachedKafkaConsumer.scala ---
@@ -62,11 +63,20 @@ private[kafka010] case class CachedKafkaConsumer private(
case class AvailableOffsetRange(earliest: Long, latest: Long)
+ private def runUninterruptiblyIfPossiable[T](body: => T): T = Thread.currentThread match {
--- End diff --
nit: rename `runUninterruptiblyIfPossible`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17761: [SPARK-20461][Core][SS]Use UninterruptibleThread for Exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17761
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17761: [SPARK-20461][Core][SS]Use UninterruptibleThread for Exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17761
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76144/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17761: [SPARK-20461][Core][SS]Use UninterruptibleThread for Exe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17761
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org