You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Burak KÖSE (JIRA)" <ji...@apache.org> on 2019/07/02 09:24:00 UTC
[jira] [Created] (SPARK-28230) Fix Spark Streaming gracefully
shutdown on Hadopp 2.8.x
Burak KÖSE created SPARK-28230:
----------------------------------
Summary: Fix Spark Streaming gracefully shutdown on Hadopp 2.8.x
Key: SPARK-28230
URL: https://issues.apache.org/jira/browse/SPARK-28230
Project: Spark
Issue Type: Bug
Components: Spark Core, Structured Streaming
Affects Versions: 2.4.3
Reporter: Burak KÖSE
Gracefully shutdown is not properly working on Hadoop 2.8.x. Hadoop introduces 10 seconds timeout by default. This is hardcoded and not something that we can configure with Hadoop settings.
[https://github.com/apache/hadoop/blob/branch-2.8.5/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ShutdownHookManager.java#L51]
However, if we use
{code:java}
public void addShutdownHook(Runnable shutdownHook, int priority, long timeout, TimeUnit unit){code}
in [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/ShutdownHookManager.scala#L180|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/ShutdownHookManager.scala#L180,]
problem might be solved.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org