You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Arun Mahadevan (JIRA)" <ji...@apache.org> on 2019/02/27 21:05:00 UTC
[jira] [Commented] (SPARK-21029) All StreamingQuery should be
stopped when the SparkSession is stopped
[ https://issues.apache.org/jira/browse/SPARK-21029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16779765#comment-16779765 ]
Arun Mahadevan commented on SPARK-21029:
----------------------------------------
I encountered this issue and propose an alternative here https://github.com/apache/spark/pull/23912
To keep it simple, this just stops the streaming queries when the spark session is stopped. It does not handle the spark context being directly stopped, since that requires more changes to add a callback hook that the spark context can invoke and the session can leverage. Using the shutdown hook like proposed previously does not look clean either.
> All StreamingQuery should be stopped when the SparkSession is stopped
> ---------------------------------------------------------------------
>
> Key: SPARK-21029
> URL: https://issues.apache.org/jira/browse/SPARK-21029
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 2.2.0, 2.3.0
> Reporter: Felix Cheung
> Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org