You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by aray <gi...@git.apache.org> on 2017/06/14 20:32:12 UTC
[GitHub] spark pull request #18306: [SPARK-21029][SS] All StreamingQuery should be st...
GitHub user aray opened a pull request:
https://github.com/apache/spark/pull/18306
[SPARK-21029][SS] All StreamingQuery should be stopped when the SparkSession is stopped
## What changes were proposed in this pull request?
Adds method to `StreamingQueryManager` that stops all active queries. Calls this method when `SparkSession.stop` is called.
## How was this patch tested?
Two unit tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aray/spark SPARK-21029
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18306.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18306
----
commit 7da14aefcaf618e9f64459d50c5f78502eeb1f59
Author: Andrew Ray <ra...@gmail.com>
Date: 2017-06-14T18:50:15Z
SPARK-21029
commit 38f9566fd6503dace04b47a4d79bc6b412e49113
Author: Andrew Ray <ra...@gmail.com>
Date: 2017-06-14T20:16:00Z
call in spark session close instead of shutdown hook
commit a64c0ff1c057dca48f0101232c433ef80d24e883
Author: Andrew Ray <ra...@gmail.com>
Date: 2017-06-14T20:29:26Z
clean imports
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18306: [SPARK-21029][SS] All StreamingQuery should be stopped w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18306
**[Test build #78725 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78725/testReport)** for PR 18306 at commit [`005472e`](https://github.com/apache/spark/commit/005472ed10fad3d1bc8feff12fc55c5682724a0e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18306: [SPARK-21029][SS] All StreamingQuery should be stopped w...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/18306
Seems okay to me. /cc @zsxwing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18306: [SPARK-21029][SS] All StreamingQuery should be stopped w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18306
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78104/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18306: [SPARK-21029][SS] All StreamingQuery should be stopped w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18306
**[Test build #78103 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78103/testReport)** for PR 18306 at commit [`4863238`](https://github.com/apache/spark/commit/4863238af3871745e0e8de45773d3ab5fafffa40).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18306: [SPARK-21029][SS] All StreamingQuery should be stopped w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18306
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18306: [SPARK-21029][SS] All StreamingQuery should be stopped w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18306
**[Test build #78103 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78103/testReport)** for PR 18306 at commit [`4863238`](https://github.com/apache/spark/commit/4863238af3871745e0e8de45773d3ab5fafffa40).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18306: [SPARK-21029][SS] All StreamingQuery should be stopped w...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/18306
Is this safe to do @marmbrus ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18306: [SPARK-21029][SS] All StreamingQuery should be stopped w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18306
**[Test build #78066 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78066/testReport)** for PR 18306 at commit [`a64c0ff`](https://github.com/apache/spark/commit/a64c0ff1c057dca48f0101232c433ef80d24e883).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18306: [SPARK-21029][SS] All StreamingQuery should be stopped w...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/18306
cc @zsxwing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18306: [SPARK-21029][SS] All StreamingQuery should be st...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/18306#discussion_r122091309
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryManager.scala ---
@@ -321,6 +321,17 @@ class StreamingQueryManager private[sql] (sparkSession: SparkSession) extends Lo
query
}
+ private[sql] def stopAllQueries(): Unit = {
+ active.foreach { query =>
+ try {
+ query.stop()
+ } catch {
+ case e: Throwable =>
--- End diff --
nit: It's better to just catch `NonFatal(e)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18306: [SPARK-21029][SS] All StreamingQuery should be st...
Posted by aray <gi...@git.apache.org>.
Github user aray commented on a diff in the pull request:
https://github.com/apache/spark/pull/18306#discussion_r136436631
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -562,6 +563,8 @@ class SparkContext(config: SparkConf) extends Logging {
}
_cleaner.foreach(_.start())
+ _stopHooks = new SparkShutdownHookManager()
--- End diff --
The queries also need to be gracefully stopped if someone calls `sc.stop()` without shutting down the JVM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18306: [SPARK-21029][SS] All StreamingQuery should be stopped w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18306
**[Test build #78104 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78104/testReport)** for PR 18306 at commit [`8cc9cbc`](https://github.com/apache/spark/commit/8cc9cbc3ed5890d0ee2d13352cfcec3df28b0aab).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18306: [SPARK-21029][SS] All StreamingQuery should be st...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/18306#discussion_r136516938
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -562,6 +563,8 @@ class SparkContext(config: SparkConf) extends Logging {
}
_cleaner.foreach(_.start())
+ _stopHooks = new SparkShutdownHookManager()
--- End diff --
I think that's my point.
if sc.stop() calls streams.stopAllQueries()
then if someone calls sc.stop() it works
if someone shutdown the JVM, sc.stop() is already getting called.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18306: [SPARK-21029][SS] All StreamingQuery should be stopped w...
Posted by aray <gi...@git.apache.org>.
Github user aray commented on the issue:
https://github.com/apache/spark/pull/18306
@zsxwing can you take another look at this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18306: [SPARK-21029][SS] All StreamingQuery should be stopped w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18306
**[Test build #78066 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78066/testReport)** for PR 18306 at commit [`a64c0ff`](https://github.com/apache/spark/commit/a64c0ff1c057dca48f0101232c433ef80d24e883).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18306: [SPARK-21029][SS] All StreamingQuery should be stopped w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18306
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18306: [SPARK-21029][SS] All StreamingQuery should be stopped w...
Posted by aray <gi...@git.apache.org>.
Github user aray commented on the issue:
https://github.com/apache/spark/pull/18306
ping @rxin @marmbrus @zsxwing @felixcheung can anyone look at this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18306: [SPARK-21029][SS] All StreamingQuery should be stopped w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18306
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78725/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18306: [SPARK-21029][SS] All StreamingQuery should be stopped w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18306
**[Test build #78104 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78104/testReport)** for PR 18306 at commit [`8cc9cbc`](https://github.com/apache/spark/commit/8cc9cbc3ed5890d0ee2d13352cfcec3df28b0aab).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18306: [SPARK-21029][SS] All StreamingQuery should be stopped w...
Posted by aray <gi...@git.apache.org>.
Github user aray commented on the issue:
https://github.com/apache/spark/pull/18306
@zsxwing @marmbrus @rxin This is ready for review. I have changed the approach so that queries from all SparkSession's are stopped. I was not able to use a SparkListener as @zsxwing suggested or even make a new event type (as I suggested) as all of that is handled asynchronously and thus queries are not stopped before the SparkContext is stopped. This new approach adds a way to register stop hooks to the SparkContext that are called in a blocking fashion on stop. Does this work for everyone?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18306: [SPARK-21029][SS] All StreamingQuery should be st...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/18306#discussion_r122092151
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala ---
@@ -690,6 +690,7 @@ class SparkSession private(
* @since 2.0.0
*/
def stop(): Unit = {
+ streams.stopAllQueries()
--- End diff --
This one actually doesn't stop queries in the child `SparkSession`s. You can register a SparkListener when creating StreamingQueryManager to monitor `SparkListenerApplicationEnd` and stop queries.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18306: [SPARK-21029][SS] All StreamingQuery should be stopped w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18306
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78103/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18306: [SPARK-21029][SS] All StreamingQuery should be stopped w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18306
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78066/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18306: [SPARK-21029][SS] All StreamingQuery should be st...
Posted by aray <gi...@git.apache.org>.
Github user aray commented on a diff in the pull request:
https://github.com/apache/spark/pull/18306#discussion_r122112073
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala ---
@@ -690,6 +690,7 @@ class SparkSession private(
* @since 2.0.0
*/
def stop(): Unit = {
+ streams.stopAllQueries()
--- End diff --
I did look into that method. It does not work because the [application end event is generated]( https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala#L1897) after the [SparkContext stopped flag is set]( https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala#L1888) which prevents the queries from stopping properly. We could add a new event type that gets called before stopped flag is set in the close method. Do you think that would be an appropriate solution?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18306: [SPARK-21029][SS] All StreamingQuery should be st...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/18306#discussion_r122122675
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryManagerSuite.scala ---
@@ -239,6 +237,40 @@ class StreamingQueryManagerSuite extends StreamTest with BeforeAndAfter {
}
}
+ test("stopAllQueries") {
+ val datasets = Seq.fill(5)(makeDataset._2)
+ withQueriesOn(datasets: _*) { queries =>
+ assert(queries.forall(_.isActive))
+ spark.streams.stopAllQueries()
+ assert(queries.forall(_.isActive == false), "Queries are still running")
+ }
+ }
+
+ test("stop session stops all queries") {
+ val inputData = MemoryStream[Int]
+ val mapped = inputData.toDS.map(6 / _)
+ var query: StreamingQuery = null
+ try {
+ query = mapped.toDF.writeStream
+ .format("memory")
+ .queryName(s"queryInNewSession")
+ .outputMode("append")
+ .start()
+ assert(query.isActive)
+ spark.stop()
+ assert(spark.sparkContext.isStopped)
+ assert(query.isActive == false, "Query is still running")
+ } catch {
+ case NonFatal(e) =>
+ if (query != null) query.stop()
+ throw e
--- End diff --
why still try/catch to stop the query? since this is a test of this specific behavior, if the query isn't stopped, or throws, the test actually is failing?
more importantly, why ignore NonFatal exception thrown?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18306: [SPARK-21029][SS] All StreamingQuery should be st...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/18306#discussion_r136410312
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -562,6 +563,8 @@ class SparkContext(config: SparkConf) extends Logging {
}
_cleaner.foreach(_.start())
+ _stopHooks = new SparkShutdownHookManager()
--- End diff --
there's already a shutdown hook to call sc.stop() - perhaps just add the clean up in stop()
https://github.com/aray/spark/blob/005472ed10fad3d1bc8feff12fc55c5682724a0e/core/src/main/scala/org/apache/spark/SparkContext.scala#L584
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18306: [SPARK-21029][SS] All StreamingQuery should be stopped w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18306
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18306: [SPARK-21029][SS] All StreamingQuery should be stopped w...
Posted by aray <gi...@git.apache.org>.
Github user aray commented on the issue:
https://github.com/apache/spark/pull/18306
ping @zsxwing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18306: [SPARK-21029][SS] All StreamingQuery should be st...
Posted by aray <gi...@git.apache.org>.
Github user aray commented on a diff in the pull request:
https://github.com/apache/spark/pull/18306#discussion_r143345713
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -562,6 +563,8 @@ class SparkContext(config: SparkConf) extends Logging {
}
_cleaner.foreach(_.start())
+ _stopHooks = new SparkShutdownHookManager()
--- End diff --
@felixcheung I don't get the objection. This is not adding a new shutdown hook, just reusing the class to add "stop hooks" that are called by `sc.stop()`. Furthermore this is needed because `sc.stop()` can not just call `stopAllQueries` unassisted without creating a circular package dependency.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18306: [SPARK-21029][SS] All StreamingQuery should be stopped w...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/18306
ping @zsxwing
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18306: [SPARK-21029][SS] All StreamingQuery should be stopped w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18306
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18306: [SPARK-21029][SS] All StreamingQuery should be stopped w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18306
**[Test build #78725 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78725/testReport)** for PR 18306 at commit [`005472e`](https://github.com/apache/spark/commit/005472ed10fad3d1bc8feff12fc55c5682724a0e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org