You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by zsxwing <gi...@git.apache.org> on 2017/11/28 22:04:37 UTC
[GitHub] spark pull request #19838: [SPARK-22638][SS]Use a separate query for Streami...
GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/19838
[SPARK-22638][SS]Use a separate query for StreamingQueryListenerBus
## What changes were proposed in this pull request?
Use a separate Spark event query for StreamingQueryListenerBus so that if there are many non-streaming events, streaming query listeners don't need to wait for other Spark listeners and can catch up.
## How was this patch tested?
Jenkins
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zsxwing/spark SPARK-22638
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19838.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19838
----
commit 60035fa865fd85cc7e9441d2dc55d46693b16dee
Author: Shixiong Zhu <zs...@gmail.com>
Date: 2017-11-28T22:03:50Z
Use a separate query for StreamingQueryListenerBus
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19838: [SPARK-22638][SS]Use a separate query for Streami...
Posted by brkyvz <gi...@git.apache.org>.
Github user brkyvz commented on a diff in the pull request:
https://github.com/apache/spark/pull/19838#discussion_r153910748
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingQueryListenerBus.scala ---
@@ -40,7 +40,7 @@ class StreamingQueryListenerBus(sparkListenerBus: LiveListenerBus)
import StreamingQueryListener._
- sparkListenerBus.addToSharedQueue(this)
+ sparkListenerBus.addToQueue(this, "streams")
--- End diff --
nit: wanna make this a constant?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19838: [SPARK-22638][SS]Use a separate query for StreamingQuery...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19838
Don't you mean "thread" instead of "query" in your summary?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19838: [SPARK-22638][SS]Use a separate query for Streami...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/19838#discussion_r153908979
--- Diff: core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala ---
@@ -87,7 +87,9 @@ private[spark] class LiveListenerBus(conf: SparkConf) {
* of each other (each one uses a separate thread for delivering events), allowing slower
* listeners to be somewhat isolated from others.
*/
- private def addToQueue(listener: SparkListenerInterface, queue: String): Unit = synchronized {
+ private[spark] def addToQueue(
--- End diff --
Change it in order to call it in StreamingQueryListenerBus.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19838: [SPARK-22638][SS]Use a separate queue for StreamingQuery...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19838
**[Test build #84319 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84319/testReport)** for PR 19838 at commit [`9b4ce99`](https://github.com/apache/spark/commit/9b4ce99baf391cf1033fc24553ca803ad86e71d2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19838: [SPARK-22638][SS]Use a separate query for StreamingQuery...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19838
**[Test build #84274 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84274/testReport)** for PR 19838 at commit [`60035fa`](https://github.com/apache/spark/commit/60035fa865fd85cc7e9441d2dc55d46693b16dee).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19838: [SPARK-22638][SS]Use a separate queue for StreamingQuery...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19838
**[Test build #84319 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84319/testReport)** for PR 19838 at commit [`9b4ce99`](https://github.com/apache/spark/commit/9b4ce99baf391cf1033fc24553ca803ad86e71d2).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19838: [SPARK-22638][SS]Use a separate queue for Streami...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing closed the pull request at:
https://github.com/apache/spark/pull/19838
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19838: [SPARK-22638][SS]Use a separate query for Streami...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19838#discussion_r153674694
--- Diff: core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala ---
@@ -87,7 +87,9 @@ private[spark] class LiveListenerBus(conf: SparkConf) {
* of each other (each one uses a separate thread for delivering events), allowing slower
* listeners to be somewhat isolated from others.
*/
- private def addToQueue(listener: SparkListenerInterface, queue: String): Unit = synchronized {
+ private[spark] def addToQueue(
--- End diff --
Is it necessary to make this change?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19838: [SPARK-22638][SS]Use a separate queue for StreamingQuery...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19838
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84319/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19838: [SPARK-22638][SS]Use a separate queue for StreamingQuery...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/19838
@vanzin yeah. Good catch. Just a typo.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19838: [SPARK-22638][SS]Use a separate query for StreamingQuery...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19838
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19838: [SPARK-22638][SS]Use a separate query for StreamingQuery...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19838
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84274/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19838: [SPARK-22638][SS]Use a separate query for StreamingQuery...
Posted by brkyvz <gi...@git.apache.org>.
Github user brkyvz commented on the issue:
https://github.com/apache/spark/pull/19838
LGTM but I have very limited context on this codepath. Maybe @tdas can also take a very quick look?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19838: [SPARK-22638][SS]Use a separate queue for StreamingQuery...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19838
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19838: [SPARK-22638][SS]Use a separate queue for StreamingQuery...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/19838
Thanks! Merging to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19838: [SPARK-22638][SS]Use a separate query for StreamingQuery...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19838
**[Test build #84274 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84274/testReport)** for PR 19838 at commit [`60035fa`](https://github.com/apache/spark/commit/60035fa865fd85cc7e9441d2dc55d46693b16dee).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19838: [SPARK-22638][SS]Use a separate queue for StreamingQuery...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/19838
LGTM.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org