You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by gaborgsomogyi <gi...@git.apache.org> on 2018/11/20 12:48:46 UTC
[GitHub] spark pull request #23095: [SPARK-23886][SS] Update query status for Continu...
GitHub user gaborgsomogyi opened a pull request:
https://github.com/apache/spark/pull/23095
[SPARK-23886][SS] Update query status for ContinuousExecution
## What changes were proposed in this pull request?
Added query status updates to ContinuousExecution.
## How was this patch tested?
Existing unit tests + added ContinuousQueryStatusAndProgressSuite.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gaborgsomogyi/spark SPARK-23886
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/23095.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #23095
----
commit 5b743abcaeea010e166d20f849af99f3ad5b842a
Author: Gabor Somogyi <ga...@...>
Date: 2018-11-19T16:02:29Z
[SPARK-23886][SS] Update query status for ContinuousExecution
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23095: [SPARK-23886][SS] Update query status for ContinuousExec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23095
**[Test build #99056 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99056/testReport)** for PR 23095 at commit [`5b743ab`](https://github.com/apache/spark/commit/5b743abcaeea010e166d20f849af99f3ad5b842a).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23095: [SPARK-23886][SS] Update query status for ContinuousExec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23095
**[Test build #99056 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99056/testReport)** for PR 23095 at commit [`5b743ab`](https://github.com/apache/spark/commit/5b743abcaeea010e166d20f849af99f3ad5b842a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23095: [SPARK-23886][SS] Update query status for ContinuousExec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23095
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #23095: [SPARK-23886][SS] Update query status for Continu...
Posted by gaborgsomogyi <gi...@git.apache.org>.
Github user gaborgsomogyi commented on a diff in the pull request:
https://github.com/apache/spark/pull/23095#discussion_r235781694
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousExecution.scala ---
@@ -117,6 +117,7 @@ class ContinuousExecution(
// For at least once, we can just ignore those reports and risk duplicates.
commitLog.getLatest() match {
case Some((latestEpochId, _)) =>
+ updateStatusMessage(s"Getting offsets from latest epoch $latestEpochId")
--- End diff --
Fixed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23095: [SPARK-23886][SS] Update query status for ContinuousExec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23095
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23095: [SPARK-23886][SS] Update query status for ContinuousExec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23095
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99192/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23095: [SPARK-23886][SS] Update query status for ContinuousExec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23095
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23095: [SPARK-23886][SS] Update query status for ContinuousExec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23095
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99071/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23095: [SPARK-23886][SS] Update query status for ContinuousExec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23095
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99056/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23095: [SPARK-23886][SS] Update query status for ContinuousExec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23095
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #23095: [SPARK-23886][SS] Update query status for Continu...
Posted by jose-torres <gi...@git.apache.org>.
Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/23095#discussion_r235473858
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousExecution.scala ---
@@ -117,6 +117,7 @@ class ContinuousExecution(
// For at least once, we can just ignore those reports and risk duplicates.
commitLog.getLatest() match {
case Some((latestEpochId, _)) =>
+ updateStatusMessage(s"Getting offsets from latest epoch $latestEpochId")
--- End diff --
nit: I'd mention that we're restarting the streaming query from an existing epoch in this message, to contrast with "starting new query" in the other branch
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23095: [SPARK-23886][SS] Update query status for ContinuousExec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23095
**[Test build #99192 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99192/testReport)** for PR 23095 at commit [`86539d0`](https://github.com/apache/spark/commit/86539d0e226f8e053935639c1bb1358787acd006).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23095: [SPARK-23886][SS] Update query status for ContinuousExec...
Posted by gaborgsomogyi <gi...@git.apache.org>.
Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/23095
cc @tdas @zsxwing
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23095: [SPARK-23886][SS] Update query status for ContinuousExec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23095
**[Test build #99071 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99071/testReport)** for PR 23095 at commit [`44b3c91`](https://github.com/apache/spark/commit/44b3c91094f218e1ec05c8e15b0fdcc400815caf).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23095: [SPARK-23886][SS] Update query status for ContinuousExec...
Posted by gaborgsomogyi <gi...@git.apache.org>.
Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/23095
cc @jose-torres @HeartSaVioR @attilapiros
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23095: [SPARK-23886][SS] Update query status for ContinuousExec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23095
**[Test build #99192 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99192/testReport)** for PR 23095 at commit [`86539d0`](https://github.com/apache/spark/commit/86539d0e226f8e053935639c1bb1358787acd006).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23095: [SPARK-23886][SS] Update query status for ContinuousExec...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23095
**[Test build #99071 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99071/testReport)** for PR 23095 at commit [`44b3c91`](https://github.com/apache/spark/commit/44b3c91094f218e1ec05c8e15b0fdcc400815caf).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #23095: [SPARK-23886][SS] Update query status for Continu...
Posted by gaborgsomogyi <gi...@git.apache.org>.
Github user gaborgsomogyi commented on a diff in the pull request:
https://github.com/apache/spark/pull/23095#discussion_r235108980
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryStatus.scala ---
@@ -28,9 +28,10 @@ import org.apache.spark.annotation.InterfaceStability
* Reports information about the instantaneous status of a streaming query.
*
* @param message A human readable description of what the stream is currently doing.
- * @param isDataAvailable True when there is new data to be processed.
+ * @param isDataAvailable True when there is new data to be processed. Doesn't apply
+ * to ContinuousExecution and always false.
--- End diff --
Changed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23095: [SPARK-23886][SS] Update query status for ContinuousExec...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23095
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #23095: [SPARK-23886][SS] Update query status for Continu...
Posted by attilapiros <gi...@git.apache.org>.
Github user attilapiros commented on a diff in the pull request:
https://github.com/apache/spark/pull/23095#discussion_r235091764
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryStatus.scala ---
@@ -28,9 +28,10 @@ import org.apache.spark.annotation.InterfaceStability
* Reports information about the instantaneous status of a streaming query.
*
* @param message A human readable description of what the stream is currently doing.
- * @param isDataAvailable True when there is new data to be processed.
+ * @param isDataAvailable True when there is new data to be processed. Doesn't apply
+ * to ContinuousExecution and always false.
--- End diff --
"and always false" => where it is always false.
(Same for a few lines below)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org