You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by zsxwing <gi...@git.apache.org> on 2016/05/26 20:37:28 UTC
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/13335
[SPARK-15580][SQL]Add ContinuousQueryInfo to make ContinuousQueryListener events serializable
## What changes were proposed in this pull request?
This PR adds ContinuousQueryInfo to make ContinuousQueryListener events serializable in order to support writing events into the event log.
## How was this patch tested?
Jenkins unit tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zsxwing/spark query-info
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13335.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13335
----
commit 4f7ee7fb68050bc16529649bb4e8337929931b3c
Author: Shixiong Zhu <sh...@databricks.com>
Date: 2016-05-26T20:33:30Z
Add ContinuousQueryInfo to make ContinuousQueryListener events serializable
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r65410405
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ContinuousQueryListenerBus.scala ---
@@ -71,15 +70,15 @@ class ContinuousQueryListenerBus(sparkListenerBus: LiveListenerBus)
}
}
- /**
- * Wrapper for StreamingListenerEvent as SparkListenerEvent so that it can be posted to Spark
- * listener bus.
- */
- private case class WrappedContinuousQueryListenerEvent(
- streamingListenerEvent: ContinuousQueryListener.Event)
- extends SparkListenerEvent {
+}
- // Do not log streaming events in event log as history server does not support these events.
- protected[spark] override def logEvent: Boolean = false
- }
+/**
+ * Wrapper for StreamingListenerEvent as SparkListenerEvent so that it can be posted to Spark
+ * listener bus.
+ */
+case class WrappedContinuousQueryListenerEvent(
+ streamingListenerEvent: ContinuousQueryListener.Event) extends SparkListenerEvent {
+
+ // Do not log streaming events in event log as history server does not support these events.
--- End diff --
It's robust. Just don't log unnecessary events since we don't have any UI for them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13335#issuecomment-222208172
**[Test build #59502 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59502/consoleFull)** for PR 13335 at commit [`e6bb76c`](https://github.com/apache/spark/commit/e6bb76ce71f75ea3e3a58d5a1f30f6a7ed06ecf3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/13335
@marmbrus could you take another look, please?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r64953837
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/util/ContinuousQueryListener.scala ---
@@ -65,11 +65,23 @@ object ContinuousQueryListener {
trait Event
/** Event representing the start of a query */
- class QueryStarted private[sql](val query: ContinuousQuery) extends Event
+ class QueryStarted private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/** Event representing any progress updates in a query */
- class QueryProgress private[sql](val query: ContinuousQuery) extends Event
+ class QueryProgress private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/** Event representing that termination of a query */
- class QueryTerminated private[sql](val query: ContinuousQuery) extends Event
+ class QueryTerminated private[sql](val queryInfo: ContinuousQueryInfo) extends Event
}
+
+/**
+ * :: Experimental ::
+ * A class that contains information about [[ContinuousQuery]].
+ */
+@Experimental
+case class ContinuousQueryInfo(
--- End diff --
We also probably don't want this to be a `case class`. The issue is that if we add new parameters in the future, the `unapply` method will break binary compatibility. The constructor should also be private.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r65962502
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/ContinuousQueryListener.scala ---
@@ -70,26 +71,34 @@ abstract class ContinuousQueryListener {
object ContinuousQueryListener {
/**
- * Base type of [[ContinuousQueryListener]] events.
+ * Base type of [[ContinuousQueryListener]] events
* @since 2.0.0
*/
- trait Event
+ trait Event extends SparkListenerEvent
/**
- * Event representing the start of a query.
+ * Event representing the start of a query
* @since 2.0.0
*/
- class QueryStarted private[sql](val query: ContinuousQuery) extends Event
+ class QueryStarted private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/**
- * Event representing any progress updates in a query.
+ * Event representing any progress updates in a query
* @since 2.0.0
*/
- class QueryProgress private[sql](val query: ContinuousQuery) extends Event
+ class QueryProgress private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/**
- * Event representing that termination of a query.
+ * Event representing that termination of a query
+ *
+ * @param queryInfo The query info.
--- End diff --
nit: "The query info" ---> "Information about the status of the query"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13335#issuecomment-222249486
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r64967228
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/util/ContinuousQueryListener.scala ---
@@ -65,11 +65,33 @@ object ContinuousQueryListener {
trait Event
/** Event representing the start of a query */
- class QueryStarted private[sql](val query: ContinuousQuery) extends Event
+ class QueryStarted private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/** Event representing any progress updates in a query */
- class QueryProgress private[sql](val query: ContinuousQuery) extends Event
+ class QueryProgress private[sql](val queryInfo: ContinuousQueryInfo) extends Event
- /** Event representing that termination of a query */
- class QueryTerminated private[sql](val query: ContinuousQuery) extends Event
+ /**
+ * Event representing that termination of a query
+ *
+ * @param queryInfo
+ * @param exception The excpetion information of the [[ContinuousQuery]] if any. Otherwise, it
+ * will be `None`.
+ */
+ class QueryTerminated private[sql](
+ val queryInfo: ContinuousQueryInfo,
+ val exception: Option[String]) extends Event
}
+
+/**
+ * :: Experimental ::
+ * A class that contains information about [[ContinuousQuery]].
+ *
+ * @param name The [[ContinuousQuery]] name
+ * @param sourceStatuses The current statuses of the [[ContinuousQuery]]'s sources.
+ * @param sinkStatus The current status of the [[ContinuousQuery]]'s sink.
+ */
+@Experimental
+class ContinuousQueryInfo private[sql](
--- End diff --
Mostly I'm thinking you will want some way to go from this to the actual query object, but maybe thats not a requirement (or could still be added later).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13335
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59752/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/13335#issuecomment-222207964
/cc @marmbrus @tdas
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13335
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r64954076
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/util/ContinuousQueryListener.scala ---
@@ -65,11 +65,23 @@ object ContinuousQueryListener {
trait Event
/** Event representing the start of a query */
- class QueryStarted private[sql](val query: ContinuousQuery) extends Event
+ class QueryStarted private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/** Event representing any progress updates in a query */
- class QueryProgress private[sql](val query: ContinuousQuery) extends Event
+ class QueryProgress private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/** Event representing that termination of a query */
- class QueryTerminated private[sql](val query: ContinuousQuery) extends Event
+ class QueryTerminated private[sql](val queryInfo: ContinuousQueryInfo) extends Event
}
+
+/**
+ * :: Experimental ::
+ * A class that contains information about [[ContinuousQuery]].
+ */
+@Experimental
+case class ContinuousQueryInfo(
+ name: String,
+ isActive: Boolean,
+ sourceStatuses: Seq[SourceStatus],
+ sinkStatus: SinkStatus,
+ exception: Option[ContinuousQueryException])
--- End diff --
Though the stacktrace is also very useful. That might be safe to serialize.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r64955737
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/util/ContinuousQueryListener.scala ---
@@ -65,11 +65,23 @@ object ContinuousQueryListener {
trait Event
/** Event representing the start of a query */
- class QueryStarted private[sql](val query: ContinuousQuery) extends Event
+ class QueryStarted private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/** Event representing any progress updates in a query */
- class QueryProgress private[sql](val query: ContinuousQuery) extends Event
+ class QueryProgress private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/** Event representing that termination of a query */
- class QueryTerminated private[sql](val query: ContinuousQuery) extends Event
+ class QueryTerminated private[sql](val queryInfo: ContinuousQueryInfo) extends Event
--- End diff --
Removed `isActive` and moved `exception` to `QueryTerminated`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r65409815
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ContinuousQueryListenerBus.scala ---
@@ -69,14 +69,15 @@ class ContinuousQueryListenerBus(sparkListenerBus: LiveListenerBus)
}
}
- /**
- * Wrapper for StreamingListenerEvent as SparkListenerEvent so that it can be posted to Spark
- * listener bus.
- */
- private case class WrappedContinuousQueryListenerEvent(
- streamingListenerEvent: ContinuousQueryListener.Event) extends SparkListenerEvent {
+}
- // Do not log streaming events in event log as history server does not support these events.
- protected[spark] override def logEvent: Boolean = false
- }
+/**
+ * Wrapper for StreamingListenerEvent as SparkListenerEvent so that it can be posted to Spark
+ * listener bus.
+ */
+case class WrappedContinuousQueryListenerEvent(
--- End diff --
I don't actually understand the structure of all the different even busses, but why do we need a different event type? Why don't all the events just extend `SparkListenerEvent`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13335
**[Test build #59760 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59760/consoleFull)** for PR 13335 at commit [`be4b8c3`](https://github.com/apache/spark/commit/be4b8c35bf8c4adfebd09cd944329b1a2f776fe0).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13335
**[Test build #60072 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60072/consoleFull)** for PR 13335 at commit [`1f00c60`](https://github.com/apache/spark/commit/1f00c608aaf8938f0f234327660d306db9b54b97).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r65966303
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/ContinuousQueryInfo.scala ---
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.streaming
+
+import org.apache.spark.annotation.Experimental
+
+/**
+ * :: Experimental ::
+ * A class used to report information about the progress of a [[ContinuousQuery]].
+ *
+ * @param name The [[ContinuousQuery]] name
--- End diff --
Super Nit: missing period after name.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13335#issuecomment-222429686
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59608/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r65410185
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/Offset.scala ---
@@ -17,11 +17,14 @@
package org.apache.spark.sql.execution.streaming
+import com.fasterxml.jackson.annotation.JsonTypeInfo
+
/**
* An offset is a monotonically increasing metric used to track progress in the computation of a
* stream. An [[Offset]] must be comparable, and the result of `compareTo` must be consistent
* with `equals` and `hashcode`.
*/
+@JsonTypeInfo(use = JsonTypeInfo.Id.CLASS, include = JsonTypeInfo.As.PROPERTY, property = "@class")
--- End diff --
What does this mean? What JSON does this produce?
@vanzin, I find this very hard to reason about relative to constructing the JSON by hand.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/13335#issuecomment-222415048
I reverted https://github.com/apache/spark/commit/c24b6b679c3efa053f7de19be73eb36dc70d9930 as there is a Option serialization bug in jackson-module-scala 2.7.3 (https://github.com/FasterXML/jackson-module-scala/issues/240)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13335
**[Test build #60072 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60072/consoleFull)** for PR 13335 at commit [`1f00c60`](https://github.com/apache/spark/commit/1f00c608aaf8938f0f234327660d306db9b54b97).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13335#issuecomment-222414970
**[Test build #59608 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59608/consoleFull)** for PR 13335 at commit [`d7c7b18`](https://github.com/apache/spark/commit/d7c7b181de77621ec15f3a7babb0619521a5554e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r64955634
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/util/ContinuousQueryListener.scala ---
@@ -65,11 +65,23 @@ object ContinuousQueryListener {
trait Event
/** Event representing the start of a query */
- class QueryStarted private[sql](val query: ContinuousQuery) extends Event
+ class QueryStarted private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/** Event representing any progress updates in a query */
- class QueryProgress private[sql](val query: ContinuousQuery) extends Event
+ class QueryProgress private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/** Event representing that termination of a query */
- class QueryTerminated private[sql](val query: ContinuousQuery) extends Event
+ class QueryTerminated private[sql](val queryInfo: ContinuousQueryInfo) extends Event
}
+
+/**
+ * :: Experimental ::
+ * A class that contains information about [[ContinuousQuery]].
+ */
+@Experimental
+case class ContinuousQueryInfo(
--- End diff --
Removed `case`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r65282660
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ContinuousQueryListenerBus.scala ---
@@ -69,14 +69,15 @@ class ContinuousQueryListenerBus(sparkListenerBus: LiveListenerBus)
}
}
- /**
- * Wrapper for StreamingListenerEvent as SparkListenerEvent so that it can be posted to Spark
- * listener bus.
- */
- private case class WrappedContinuousQueryListenerEvent(
- streamingListenerEvent: ContinuousQueryListener.Event) extends SparkListenerEvent {
+}
- // Do not log streaming events in event log as history server does not support these events.
- protected[spark] override def logEvent: Boolean = false
- }
+/**
+ * Wrapper for StreamingListenerEvent as SparkListenerEvent so that it can be posted to Spark
+ * listener bus.
+ */
+case class WrappedContinuousQueryListenerEvent(
--- End diff --
Move it out of `ContinuousQueryListenerBus` in order to eliminate the reference
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r64960049
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/util/ContinuousQueryListener.scala ---
@@ -65,11 +65,33 @@ object ContinuousQueryListener {
trait Event
/** Event representing the start of a query */
- class QueryStarted private[sql](val query: ContinuousQuery) extends Event
+ class QueryStarted private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/** Event representing any progress updates in a query */
- class QueryProgress private[sql](val query: ContinuousQuery) extends Event
+ class QueryProgress private[sql](val queryInfo: ContinuousQueryInfo) extends Event
- /** Event representing that termination of a query */
- class QueryTerminated private[sql](val query: ContinuousQuery) extends Event
+ /**
+ * Event representing that termination of a query
+ *
+ * @param queryInfo
+ * @param exception The excpetion information of the [[ContinuousQuery]] if any. Otherwise, it
+ * will be `None`.
+ */
+ class QueryTerminated private[sql](
+ val queryInfo: ContinuousQueryInfo,
+ val exception: Option[String]) extends Event
}
+
+/**
+ * :: Experimental ::
+ * A class that contains information about [[ContinuousQuery]].
--- End diff --
A class used to report information about the progress of a [[ContinuousQuery]].
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r64953741
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/util/ContinuousQueryListener.scala ---
@@ -65,11 +65,23 @@ object ContinuousQueryListener {
trait Event
/** Event representing the start of a query */
- class QueryStarted private[sql](val query: ContinuousQuery) extends Event
+ class QueryStarted private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/** Event representing any progress updates in a query */
- class QueryProgress private[sql](val query: ContinuousQuery) extends Event
+ class QueryProgress private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/** Event representing that termination of a query */
- class QueryTerminated private[sql](val query: ContinuousQuery) extends Event
+ class QueryTerminated private[sql](val queryInfo: ContinuousQueryInfo) extends Event
--- End diff --
Should these be different events? Its a little odd that `Progress` includes and `exception` and `Terminated` includes `isActive`. We should consider what is the minimal amount of data we can expose here such that you can do all the useful things we need to.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r64959957
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/util/ContinuousQueryListener.scala ---
@@ -65,11 +65,33 @@ object ContinuousQueryListener {
trait Event
/** Event representing the start of a query */
- class QueryStarted private[sql](val query: ContinuousQuery) extends Event
+ class QueryStarted private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/** Event representing any progress updates in a query */
- class QueryProgress private[sql](val query: ContinuousQuery) extends Event
+ class QueryProgress private[sql](val queryInfo: ContinuousQueryInfo) extends Event
- /** Event representing that termination of a query */
- class QueryTerminated private[sql](val query: ContinuousQuery) extends Event
+ /**
+ * Event representing that termination of a query
+ *
+ * @param queryInfo
+ * @param exception The excpetion information of the [[ContinuousQuery]] if any. Otherwise, it
--- End diff --
`exception`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13335#issuecomment-222249487
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59514/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13335
**[Test build #59752 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59752/consoleFull)** for PR 13335 at commit [`7df7e9c`](https://github.com/apache/spark/commit/7df7e9c2b729ebf4ad2b4941293884e0643b53e9).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13335
**[Test build #59752 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59752/consoleFull)** for PR 13335 at commit [`7df7e9c`](https://github.com/apache/spark/commit/7df7e9c2b729ebf4ad2b4941293884e0643b53e9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13335
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59771/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r65411829
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/Offset.scala ---
@@ -17,11 +17,14 @@
package org.apache.spark.sql.execution.streaming
+import com.fasterxml.jackson.annotation.JsonTypeInfo
+
/**
* An offset is a monotonically increasing metric used to track progress in the computation of a
* stream. An [[Offset]] must be comparable, and the result of `compareTo` must be consistent
* with `equals` and `hashcode`.
*/
+@JsonTypeInfo(use = JsonTypeInfo.Id.CLASS, include = JsonTypeInfo.As.PROPERTY, property = "@class")
--- End diff --
In my experience (with jackson specifically and people who have read the docs) you end up with surprises down the road due to magic.
@zsxwing In this particular case, maybe we should remove the offsets from the status messages, or turn the to strings or something. These are going to be implemented by specific sinks, so I'm not sure we want to expose the details of arbitrary classes to the listener bus.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/13335
> I reverted c24b6b6 as there is a Option serialization bug in jackson-module-scala 2.7.3 (FasterXML/jackson-module-scala#240)
Removed this one and put it in https://github.com/apache/spark/pull/13417. Will update this one once #13417 gets merged
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13335
**[Test build #59758 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59758/consoleFull)** for PR 13335 at commit [`00ccfdf`](https://github.com/apache/spark/commit/00ccfdf76dab6baffdbb88be498bc8665143529f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/13335
LGTM. Will merge after tests pass.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r64958406
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/util/ContinuousQueryListenerSuite.scala ---
@@ -183,35 +179,23 @@ class ContinuousQueryListenerSuite extends StreamTest with SharedSQLContext with
override def onQueryStarted(queryStarted: QueryStarted): Unit = {
asyncTestWaiter {
- startStatus = QueryStatus(queryStarted.query)
+ startStatus = queryStarted.queryInfo
}
}
override def onQueryProgress(queryProgress: QueryProgress): Unit = {
asyncTestWaiter {
assert(startStatus != null, "onQueryProgress called before onQueryStarted")
- progressStatuses.add(QueryStatus(queryProgress.query))
+ progressStatuses.add(queryProgress.queryInfo)
}
}
override def onQueryTerminated(queryTerminated: QueryTerminated): Unit = {
asyncTestWaiter {
assert(startStatus != null, "onQueryTerminated called before onQueryStarted")
- terminationStatus = QueryStatus(queryTerminated.query)
+ terminationStatus = queryTerminated.queryInfo
--- End diff --
The exception is not tested.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13335#issuecomment-221988548
**[Test build #59415 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59415/consoleFull)** for PR 13335 at commit [`4f7ee7f`](https://github.com/apache/spark/commit/4f7ee7fb68050bc16529649bb4e8337929931b3c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r64858840
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala ---
@@ -484,6 +484,15 @@ class StreamExecution(
""".stripMargin
}
+ private def toInfo: ContinuousQueryInfo = {
--- End diff --
Do we need to make this a deep copy, especially for `sourceStatuses` and `sinkStatus` ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13335
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59674/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r65409474
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/ContinuousQueryListener.scala ---
@@ -69,27 +72,27 @@ abstract class ContinuousQueryListener {
@Experimental
object ContinuousQueryListener {
- /**
- * Base type of [[ContinuousQueryListener]] events.
- * @since 2.0.0
- */
+ /** Base type of [[ContinuousQueryListener]] events */
+ @JsonTypeInfo(
+ use = JsonTypeInfo.Id.CLASS,
+ include = JsonTypeInfo.As.PROPERTY,
+ property = "@class")
trait Event
- /**
- * Event representing the start of a query.
- * @since 2.0.0
- */
- class QueryStarted private[sql](val query: ContinuousQuery) extends Event
+ /** Event representing the start of a query */
+ class QueryStarted private[sql](val queryInfo: ContinuousQueryInfo) extends Event
- /**
- * Event representing any progress updates in a query.
- * @since 2.0.0
- */
- class QueryProgress private[sql](val query: ContinuousQuery) extends Event
+ /** Event representing any progress updates in a query */
+ class QueryProgress private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/**
- * Event representing that termination of a query.
- * @since 2.0.0
+ * Event representing that termination of a query
+ *
+ * @param queryInfo
+ * @param exception The exception information of the [[ContinuousQuery]] if any. Otherwise, it
+ * will be `None`.
*/
- class QueryTerminated private[sql](val query: ContinuousQuery) extends Event
+ class QueryTerminated private[sql](
+ val queryInfo: ContinuousQueryInfo,
+ val exception: Option[String]) extends Event
--- End diff --
I don't think a string is enough here. I think you need the stack trace also for it to be useful.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13335
**[Test build #60069 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60069/consoleFull)** for PR 13335 at commit [`3230a12`](https://github.com/apache/spark/commit/3230a1295c115eaa29be3beae57efc8a9abad2bd).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13335
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/13335
Merging this to master and 2.0. Thanks @zsxwing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13335#issuecomment-222012409
**[Test build #59415 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59415/consoleFull)** for PR 13335 at commit [`4f7ee7f`](https://github.com/apache/spark/commit/4f7ee7fb68050bc16529649bb4e8337929931b3c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class ContinuousQueryInfo(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r65965736
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/ContinuousQueryInfo.scala ---
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.streaming
+
+import org.apache.spark.annotation.Experimental
+
+/**
+ * :: Experimental ::
+ * A class used to report information about the progress of a [[ContinuousQuery]].
+ *
+ * @param name The [[ContinuousQuery]] name
+ * @param sourceStatuses The current statuses of the [[ContinuousQuery]]'s sources.
+ * @param sinkStatus The current status of the [[ContinuousQuery]]'s sink.
+ */
+@Experimental
+class ContinuousQueryInfo private[sql](
+ val name: String,
+ val sourceStatuses: Seq[SourceStatus],
--- End diff --
This should also probably be an Array. No need to be a Seq.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13335
**[Test build #59771 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59771/consoleFull)** for PR 13335 at commit [`8dfc105`](https://github.com/apache/spark/commit/8dfc10524bc3407ff640c9fd1d03d3d13f1c6e50).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13335
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60069/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/13335
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13335#issuecomment-222231901
**[Test build #59502 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59502/consoleFull)** for PR 13335 at commit [`e6bb76c`](https://github.com/apache/spark/commit/e6bb76ce71f75ea3e3a58d5a1f30f6a7ed06ecf3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13335#issuecomment-222429496
**[Test build #59608 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59608/consoleFull)** for PR 13335 at commit [`d7c7b18`](https://github.com/apache/spark/commit/d7c7b181de77621ec15f3a7babb0619521a5554e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` * case class Person(name: String, age: Long)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13335
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59685/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r64966575
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/util/ContinuousQueryListener.scala ---
@@ -65,11 +65,33 @@ object ContinuousQueryListener {
trait Event
/** Event representing the start of a query */
- class QueryStarted private[sql](val query: ContinuousQuery) extends Event
+ class QueryStarted private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/** Event representing any progress updates in a query */
- class QueryProgress private[sql](val query: ContinuousQuery) extends Event
+ class QueryProgress private[sql](val queryInfo: ContinuousQueryInfo) extends Event
- /** Event representing that termination of a query */
- class QueryTerminated private[sql](val query: ContinuousQuery) extends Event
+ /**
+ * Event representing that termination of a query
+ *
+ * @param queryInfo
+ * @param exception The excpetion information of the [[ContinuousQuery]] if any. Otherwise, it
+ * will be `None`.
+ */
+ class QueryTerminated private[sql](
+ val queryInfo: ContinuousQueryInfo,
+ val exception: Option[String]) extends Event
}
+
+/**
+ * :: Experimental ::
+ * A class that contains information about [[ContinuousQuery]].
+ *
+ * @param name The [[ContinuousQuery]] name
+ * @param sourceStatuses The current statuses of the [[ContinuousQuery]]'s sources.
+ * @param sinkStatus The current status of the [[ContinuousQuery]]'s sink.
+ */
+@Experimental
+class ContinuousQueryInfo private[sql](
--- End diff --
The name is unique within each session i believe. Maybe we should have a string representing the session (maybe sessionState.toString for start)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13335#issuecomment-222012654
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59415/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r65964733
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/ContinuousQueryListener.scala ---
@@ -70,26 +71,34 @@ abstract class ContinuousQueryListener {
object ContinuousQueryListener {
/**
- * Base type of [[ContinuousQueryListener]] events.
+ * Base type of [[ContinuousQueryListener]] events
* @since 2.0.0
*/
- trait Event
+ trait Event extends SparkListenerEvent
/**
- * Event representing the start of a query.
+ * Event representing the start of a query
* @since 2.0.0
*/
- class QueryStarted private[sql](val query: ContinuousQuery) extends Event
+ class QueryStarted private[sql](val queryInfo: ContinuousQueryInfo) extends Event
--- End diff --
Done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13335
**[Test build #59674 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59674/consoleFull)** for PR 13335 at commit [`3669df2`](https://github.com/apache/spark/commit/3669df28f27082364f4b67f2fef18861fc6ca514).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class CompositeOffset(offsets: Array[Option[Offset]]) extends Offset `
* `case class WrappedContinuousQueryListenerEvent(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13335
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13335
**[Test build #59760 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59760/consoleFull)** for PR 13335 at commit [`be4b8c3`](https://github.com/apache/spark/commit/be4b8c35bf8c4adfebd09cd944329b1a2f776fe0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r65024904
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/util/ContinuousQueryListenerSuite.scala ---
@@ -183,35 +179,23 @@ class ContinuousQueryListenerSuite extends StreamTest with SharedSQLContext with
override def onQueryStarted(queryStarted: QueryStarted): Unit = {
asyncTestWaiter {
- startStatus = QueryStatus(queryStarted.query)
+ startStatus = queryStarted.queryInfo
}
}
override def onQueryProgress(queryProgress: QueryProgress): Unit = {
asyncTestWaiter {
assert(startStatus != null, "onQueryProgress called before onQueryStarted")
- progressStatuses.add(QueryStatus(queryProgress.query))
+ progressStatuses.add(queryProgress.queryInfo)
}
}
override def onQueryTerminated(queryTerminated: QueryTerminated): Unit = {
asyncTestWaiter {
assert(startStatus != null, "onQueryTerminated called before onQueryStarted")
- terminationStatus = QueryStatus(queryTerminated.query)
+ terminationStatus = queryTerminated.queryInfo
--- End diff --
Added a test for the exception
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r64960270
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/util/ContinuousQueryListener.scala ---
@@ -65,11 +65,33 @@ object ContinuousQueryListener {
trait Event
/** Event representing the start of a query */
- class QueryStarted private[sql](val query: ContinuousQuery) extends Event
+ class QueryStarted private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/** Event representing any progress updates in a query */
- class QueryProgress private[sql](val query: ContinuousQuery) extends Event
+ class QueryProgress private[sql](val queryInfo: ContinuousQueryInfo) extends Event
- /** Event representing that termination of a query */
- class QueryTerminated private[sql](val query: ContinuousQuery) extends Event
+ /**
+ * Event representing that termination of a query
+ *
+ * @param queryInfo
+ * @param exception The excpetion information of the [[ContinuousQuery]] if any. Otherwise, it
+ * will be `None`.
+ */
+ class QueryTerminated private[sql](
+ val queryInfo: ContinuousQueryInfo,
+ val exception: Option[String]) extends Event
--- End diff --
I think we want the stacktrace too as well as how far it made it before it failed. It might be okay to serialize the exception itself, we just need to make sure its not pulling in the whole query.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13335
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60072/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13335#issuecomment-222233618
**[Test build #59514 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59514/consoleFull)** for PR 13335 at commit [`621a867`](https://github.com/apache/spark/commit/621a8672d574d8c05774d930465f4523c77695cc).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13335#issuecomment-222249262
**[Test build #59514 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59514/consoleFull)** for PR 13335 at commit [`621a867`](https://github.com/apache/spark/commit/621a8672d574d8c05774d930465f4523c77695cc).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r64953763
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/util/ContinuousQueryListener.scala ---
@@ -65,11 +65,23 @@ object ContinuousQueryListener {
trait Event
/** Event representing the start of a query */
- class QueryStarted private[sql](val query: ContinuousQuery) extends Event
+ class QueryStarted private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/** Event representing any progress updates in a query */
- class QueryProgress private[sql](val query: ContinuousQuery) extends Event
+ class QueryProgress private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/** Event representing that termination of a query */
- class QueryTerminated private[sql](val query: ContinuousQuery) extends Event
+ class QueryTerminated private[sql](val queryInfo: ContinuousQueryInfo) extends Event
}
+
+/**
+ * :: Experimental ::
+ * A class that contains information about [[ContinuousQuery]].
+ */
+@Experimental
+case class ContinuousQueryInfo(
+ name: String,
+ isActive: Boolean,
+ sourceStatuses: Seq[SourceStatus],
+ sinkStatus: SinkStatus,
+ exception: Option[ContinuousQueryException])
--- End diff --
Good catch. We should use `string` instead of `ContinuousQueryException`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r65963896
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/ContinuousQueryListener.scala ---
@@ -70,26 +71,34 @@ abstract class ContinuousQueryListener {
object ContinuousQueryListener {
/**
- * Base type of [[ContinuousQueryListener]] events.
+ * Base type of [[ContinuousQueryListener]] events
* @since 2.0.0
*/
- trait Event
+ trait Event extends SparkListenerEvent
/**
- * Event representing the start of a query.
+ * Event representing the start of a query
* @since 2.0.0
*/
- class QueryStarted private[sql](val query: ContinuousQuery) extends Event
+ class QueryStarted private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/**
- * Event representing any progress updates in a query.
+ * Event representing any progress updates in a query
* @since 2.0.0
*/
- class QueryProgress private[sql](val query: ContinuousQuery) extends Event
+ class QueryProgress private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/**
- * Event representing that termination of a query.
+ * Event representing that termination of a query
+ *
+ * @param queryInfo The query info.
+ * @param exception The exception message of the [[ContinuousQuery]] if any. Otherwise, it
+ * will be `None`.
+ * @param stackTrace The stack trace of the exception if any.
* @since 2.0.0
*/
- class QueryTerminated private[sql](val query: ContinuousQuery) extends Event
+ class QueryTerminated private[sql](
+ val queryInfo: ContinuousQueryInfo,
+ val exception: Option[String],
+ val stackTrace: Seq[StackTraceElement]) extends Event
--- End diff --
I don't want to expose a mutable Array to the user. Secondly, this one is not Java friendly already since it uses `Option`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r65970018
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/ContinuousQueryInfo.scala ---
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.streaming
+
+import org.apache.spark.annotation.Experimental
+
+/**
+ * :: Experimental ::
+ * A class used to report information about the progress of a [[ContinuousQuery]].
+ *
+ * @param name The [[ContinuousQuery]] name
+ * @param sourceStatuses The current statuses of the [[ContinuousQuery]]'s sources.
+ * @param sinkStatus The current status of the [[ContinuousQuery]]'s sink.
+ */
+@Experimental
+class ContinuousQueryInfo private[sql](
+ val name: String,
+ val sourceStatuses: Seq[SourceStatus],
--- End diff --
Offline discussion: this is okay. this is how it is for SparkListener, and avoid problems like the array being accidentally modified internally by the listener generator.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r65961978
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/SourceStatus.scala ---
@@ -31,4 +31,4 @@ import org.apache.spark.sql.execution.streaming.{Offset, Source}
@Experimental
class SourceStatus private[sql] (
val description: String,
- val offset: Option[Offset])
+ val offset: Option[String])
--- End diff --
offset --> offsetDesc, so that it allows us in future to expose actual offsets without the naming being weird.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r65282727
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/CompositeOffset.scala ---
@@ -67,6 +67,15 @@ case class CompositeOffset(offsets: Seq[Option[Offset]]) extends Offset {
override def toString: String =
offsets.map(_.map(_.toString).getOrElse("-")).mkString("[", ", ", "]")
+
+ override def equals(other: Any): Boolean = other match {
+ case that: CompositeOffset => offsets != null && offsets.sameElements(that.offsets)
--- End diff --
Need to redefine `equals` and `hashCode` as `offsets` is an array
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/13335
@tdas addressed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r64953212
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/util/ContinuousQueryListener.scala ---
@@ -65,11 +65,23 @@ object ContinuousQueryListener {
trait Event
/** Event representing the start of a query */
- class QueryStarted private[sql](val query: ContinuousQuery) extends Event
+ class QueryStarted private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/** Event representing any progress updates in a query */
- class QueryProgress private[sql](val query: ContinuousQuery) extends Event
+ class QueryProgress private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/** Event representing that termination of a query */
- class QueryTerminated private[sql](val query: ContinuousQuery) extends Event
+ class QueryTerminated private[sql](val queryInfo: ContinuousQueryInfo) extends Event
}
+
+/**
+ * :: Experimental ::
+ * A class that contains information about [[ContinuousQuery]].
+ */
+@Experimental
+case class ContinuousQueryInfo(
--- End diff --
We should scaladoc these parameters.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13335
**[Test build #59685 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59685/consoleFull)** for PR 13335 at commit [`9b539cd`](https://github.com/apache/spark/commit/9b539cd1d23909de7df2c27739d188ee484418ea).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` * case class Person(name: String, age: Long)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r64960190
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/util/ContinuousQueryListener.scala ---
@@ -65,11 +65,33 @@ object ContinuousQueryListener {
trait Event
/** Event representing the start of a query */
- class QueryStarted private[sql](val query: ContinuousQuery) extends Event
+ class QueryStarted private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/** Event representing any progress updates in a query */
- class QueryProgress private[sql](val query: ContinuousQuery) extends Event
+ class QueryProgress private[sql](val queryInfo: ContinuousQueryInfo) extends Event
- /** Event representing that termination of a query */
- class QueryTerminated private[sql](val query: ContinuousQuery) extends Event
+ /**
+ * Event representing that termination of a query
+ *
+ * @param queryInfo
+ * @param exception The excpetion information of the [[ContinuousQuery]] if any. Otherwise, it
+ * will be `None`.
+ */
+ class QueryTerminated private[sql](
+ val queryInfo: ContinuousQueryInfo,
+ val exception: Option[String]) extends Event
}
+
+/**
+ * :: Experimental ::
+ * A class that contains information about [[ContinuousQuery]].
+ *
+ * @param name The [[ContinuousQuery]] name
+ * @param sourceStatuses The current statuses of the [[ContinuousQuery]]'s sources.
+ * @param sinkStatus The current status of the [[ContinuousQuery]]'s sink.
+ */
+@Experimental
+class ContinuousQueryInfo private[sql](
--- End diff --
A few more questions:
- Is there a reason this isn't in the object with the other listener messages? At a minimum it should probably be defined near `SourceStatus` (which is currently defined at the top level). We should find a place to colocate all of them.
- Is `name` enough to uniquely identify a streaming query?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13335
**[Test build #59758 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59758/consoleFull)** for PR 13335 at commit [`00ccfdf`](https://github.com/apache/spark/commit/00ccfdf76dab6baffdbb88be498bc8665143529f).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r64940484
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala ---
@@ -484,6 +484,15 @@ class StreamExecution(
""".stripMargin
}
+ private def toInfo: ContinuousQueryInfo = {
--- End diff --
No. They are immutable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13335
**[Test build #60069 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60069/consoleFull)** for PR 13335 at commit [`3230a12`](https://github.com/apache/spark/commit/3230a1295c115eaa29be3beae57efc8a9abad2bd).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r65417137
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ContinuousQueryListenerBus.scala ---
@@ -69,14 +69,15 @@ class ContinuousQueryListenerBus(sparkListenerBus: LiveListenerBus)
}
}
- /**
- * Wrapper for StreamingListenerEvent as SparkListenerEvent so that it can be posted to Spark
- * listener bus.
- */
- private case class WrappedContinuousQueryListenerEvent(
- streamingListenerEvent: ContinuousQueryListener.Event) extends SparkListenerEvent {
+}
- // Do not log streaming events in event log as history server does not support these events.
- protected[spark] override def logEvent: Boolean = false
- }
+/**
+ * Wrapper for StreamingListenerEvent as SparkListenerEvent so that it can be posted to Spark
+ * listener bus.
+ */
+case class WrappedContinuousQueryListenerEvent(
--- End diff --
Good point. This pattern is from the Streaming events. But that's for binary compatibility. We don't need this pattern in the new APIs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13335#issuecomment-222232249
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r64953510
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/util/ContinuousQueryListener.scala ---
@@ -65,11 +65,23 @@ object ContinuousQueryListener {
trait Event
/** Event representing the start of a query */
- class QueryStarted private[sql](val query: ContinuousQuery) extends Event
+ class QueryStarted private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/** Event representing any progress updates in a query */
- class QueryProgress private[sql](val query: ContinuousQuery) extends Event
+ class QueryProgress private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/** Event representing that termination of a query */
- class QueryTerminated private[sql](val query: ContinuousQuery) extends Event
+ class QueryTerminated private[sql](val queryInfo: ContinuousQueryInfo) extends Event
}
+
+/**
+ * :: Experimental ::
+ * A class that contains information about [[ContinuousQuery]].
+ */
+@Experimental
+case class ContinuousQueryInfo(
+ name: String,
+ isActive: Boolean,
+ sourceStatuses: Seq[SourceStatus],
+ sinkStatus: SinkStatus,
+ exception: Option[ContinuousQueryException])
--- End diff --
Is `ContinuousQueryException` really safe to serialize with the default serializer? Does jackson obey `transient`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/13335#issuecomment-222242009
Can you include the json produced as a sanity check?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13335
**[Test build #59685 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59685/consoleFull)** for PR 13335 at commit [`9b539cd`](https://github.com/apache/spark/commit/9b539cd1d23909de7df2c27739d188ee484418ea).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r65966655
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/ContinuousQueryListener.scala ---
@@ -70,26 +71,34 @@ abstract class ContinuousQueryListener {
object ContinuousQueryListener {
/**
- * Base type of [[ContinuousQueryListener]] events.
+ * Base type of [[ContinuousQueryListener]] events
* @since 2.0.0
*/
- trait Event
+ trait Event extends SparkListenerEvent
/**
- * Event representing the start of a query.
+ * Event representing the start of a query
* @since 2.0.0
*/
- class QueryStarted private[sql](val query: ContinuousQuery) extends Event
+ class QueryStarted private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/**
- * Event representing any progress updates in a query.
+ * Event representing any progress updates in a query
* @since 2.0.0
*/
- class QueryProgress private[sql](val query: ContinuousQuery) extends Event
+ class QueryProgress private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/**
- * Event representing that termination of a query.
+ * Event representing that termination of a query
+ *
+ * @param queryInfo The query info.
+ * @param exception The exception message of the [[ContinuousQuery]] if any. Otherwise, it
+ * will be `None`.
+ * @param stackTrace The stack trace of the exception if any.
* @since 2.0.0
*/
- class QueryTerminated private[sql](val query: ContinuousQuery) extends Event
+ class QueryTerminated private[sql](
+ val queryInfo: ContinuousQueryInfo,
+ val exception: Option[String],
+ val stackTrace: Seq[StackTraceElement]) extends Event
--- End diff --
What mutable about this Array? And for Options, there is no workaround. And Option is pretty simple to deal with in Java. Seq is more annoying, need to look up Scala doc etc.?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13335
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r65411215
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/Offset.scala ---
@@ -17,11 +17,14 @@
package org.apache.spark.sql.execution.streaming
+import com.fasterxml.jackson.annotation.JsonTypeInfo
+
/**
* An offset is a monotonically increasing metric used to track progress in the computation of a
* stream. An [[Offset]] must be comparable, and the result of `compareTo` must be consistent
* with `equals` and `hashcode`.
*/
+@JsonTypeInfo(use = JsonTypeInfo.Id.CLASS, include = JsonTypeInfo.As.PROPERTY, property = "@class")
--- End diff --
That's why Jackson has javadocs. :-) You read it once instead of having to understand how every single type ends up in json by having to read the specific manual serialization code for that type...
Anyway, to avoid copy & paste of this kind of code everywhere, it would be probably better to have a base trait for "traits that will be serialized in events and might have multiple implementations", and have that trait have this annotation (like `SparkListenerEvent` does).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13335
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59760/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r65409940
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ContinuousQueryListenerBus.scala ---
@@ -71,15 +70,15 @@ class ContinuousQueryListenerBus(sparkListenerBus: LiveListenerBus)
}
}
- /**
- * Wrapper for StreamingListenerEvent as SparkListenerEvent so that it can be posted to Spark
- * listener bus.
- */
- private case class WrappedContinuousQueryListenerEvent(
- streamingListenerEvent: ContinuousQueryListener.Event)
- extends SparkListenerEvent {
+}
- // Do not log streaming events in event log as history server does not support these events.
- protected[spark] override def logEvent: Boolean = false
- }
+/**
+ * Wrapper for StreamingListenerEvent as SparkListenerEvent so that it can be posted to Spark
+ * listener bus.
+ */
+case class WrappedContinuousQueryListenerEvent(
+ streamingListenerEvent: ContinuousQueryListener.Event) extends SparkListenerEvent {
+
+ // Do not log streaming events in event log as history server does not support these events.
--- End diff --
The history server isn't robust to unknown messages? Seems bad that we always drop them. What happens when we do add support?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13335#issuecomment-222232252
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59502/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r65963272
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/ContinuousQueryListenerSuite.scala ---
@@ -141,6 +139,92 @@ class ContinuousQueryListenerSuite extends StreamTest with BeforeAndAfter {
}
}
+ test("exception should be reported in QueryTerminated") {
+ val listener = new QueryStatusCollector
+ withListenerAdded(listener) {
+ val input = MemoryStream[Int]
+ testStream(input.toDS.map(_ / 0))(
+ StartStream(),
+ AddData(input, 1),
+ ExpectFailure[SparkException](),
+ Assert {
+ spark.sparkContext.listenerBus.waitUntilEmpty(10000)
+ assert(listener.terminationStatus !== null)
+ assert(listener.terminationException.isDefined &&
--- End diff --
nit: make these conditions separate assert so its easy to debug if something fails.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r65412480
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/Offset.scala ---
@@ -17,11 +17,14 @@
package org.apache.spark.sql.execution.streaming
+import com.fasterxml.jackson.annotation.JsonTypeInfo
+
/**
* An offset is a monotonically increasing metric used to track progress in the computation of a
* stream. An [[Offset]] must be comparable, and the result of `compareTo` must be consistent
* with `equals` and `hashcode`.
*/
+@JsonTypeInfo(use = JsonTypeInfo.Id.CLASS, include = JsonTypeInfo.As.PROPERTY, property = "@class")
--- End diff --
I've had way more problems with manual serialization than with jackson, so well, YMMV I guess.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13335
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r65962584
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/ContinuousQueryListener.scala ---
@@ -70,26 +71,34 @@ abstract class ContinuousQueryListener {
object ContinuousQueryListener {
/**
- * Base type of [[ContinuousQueryListener]] events.
+ * Base type of [[ContinuousQueryListener]] events
* @since 2.0.0
*/
- trait Event
+ trait Event extends SparkListenerEvent
/**
- * Event representing the start of a query.
+ * Event representing the start of a query
* @since 2.0.0
*/
- class QueryStarted private[sql](val query: ContinuousQuery) extends Event
+ class QueryStarted private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/**
- * Event representing any progress updates in a query.
+ * Event representing any progress updates in a query
* @since 2.0.0
*/
- class QueryProgress private[sql](val query: ContinuousQuery) extends Event
+ class QueryProgress private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/**
- * Event representing that termination of a query.
+ * Event representing that termination of a query
+ *
+ * @param queryInfo The query info.
+ * @param exception The exception message of the [[ContinuousQuery]] if any. Otherwise, it
--- End diff --
nit: if any --> if the query was terminated with an exception.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r65967892
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/ContinuousQueryListener.scala ---
@@ -70,26 +71,34 @@ abstract class ContinuousQueryListener {
object ContinuousQueryListener {
/**
- * Base type of [[ContinuousQueryListener]] events.
+ * Base type of [[ContinuousQueryListener]] events
* @since 2.0.0
*/
- trait Event
+ trait Event extends SparkListenerEvent
/**
- * Event representing the start of a query.
+ * Event representing the start of a query
* @since 2.0.0
*/
- class QueryStarted private[sql](val query: ContinuousQuery) extends Event
+ class QueryStarted private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/**
- * Event representing any progress updates in a query.
+ * Event representing any progress updates in a query
* @since 2.0.0
*/
- class QueryProgress private[sql](val query: ContinuousQuery) extends Event
+ class QueryProgress private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/**
- * Event representing that termination of a query.
+ * Event representing that termination of a query
+ *
+ * @param queryInfo The query info.
+ * @param exception The exception message of the [[ContinuousQuery]] if any. Otherwise, it
+ * will be `None`.
+ * @param stackTrace The stack trace of the exception if any.
* @since 2.0.0
*/
- class QueryTerminated private[sql](val query: ContinuousQuery) extends Event
+ class QueryTerminated private[sql](
+ val queryInfo: ContinuousQueryInfo,
+ val exception: Option[String],
+ val stackTrace: Seq[StackTraceElement]) extends Event
--- End diff --
E.g., the user can change the content of an array.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/13335
@marmbrus addressed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r65962780
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/ContinuousQueryListener.scala ---
@@ -70,26 +71,34 @@ abstract class ContinuousQueryListener {
object ContinuousQueryListener {
/**
- * Base type of [[ContinuousQueryListener]] events.
+ * Base type of [[ContinuousQueryListener]] events
* @since 2.0.0
*/
- trait Event
+ trait Event extends SparkListenerEvent
/**
- * Event representing the start of a query.
+ * Event representing the start of a query
* @since 2.0.0
*/
- class QueryStarted private[sql](val query: ContinuousQuery) extends Event
+ class QueryStarted private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/**
- * Event representing any progress updates in a query.
+ * Event representing any progress updates in a query
* @since 2.0.0
*/
- class QueryProgress private[sql](val query: ContinuousQuery) extends Event
+ class QueryProgress private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/**
- * Event representing that termination of a query.
+ * Event representing that termination of a query
+ *
+ * @param queryInfo The query info.
+ * @param exception The exception message of the [[ContinuousQuery]] if any. Otherwise, it
+ * will be `None`.
+ * @param stackTrace The stack trace of the exception if any.
--- End diff --
nit: similar change as above. Also document that it will be empty if there was no error.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r65024979
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/CompositeOffset.scala ---
@@ -22,7 +22,7 @@ package org.apache.spark.sql.execution.streaming
* [[Source]]s that are present in a streaming query. This is similar to simplified, single-instance
* vector clock that must progress linearly forward.
*/
-case class CompositeOffset(offsets: Seq[Option[Offset]]) extends Offset {
+case class CompositeOffset(offsets: Array[Option[Offset]]) extends Offset {
--- End diff --
Change the type to `Array` as jackson-module-scala doesn't support `Seq[Option[X]]`. Probably because of the type erasure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13335
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13335
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59758/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13335#issuecomment-222429684
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13335
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r65411387
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/Offset.scala ---
@@ -17,11 +17,14 @@
package org.apache.spark.sql.execution.streaming
+import com.fasterxml.jackson.annotation.JsonTypeInfo
+
/**
* An offset is a monotonically increasing metric used to track progress in the computation of a
* stream. An [[Offset]] must be comparable, and the result of `compareTo` must be consistent
* with `equals` and `hashcode`.
*/
+@JsonTypeInfo(use = JsonTypeInfo.Id.CLASS, include = JsonTypeInfo.As.PROPERTY, property = "@class")
--- End diff --
> What does this mean? What JSON does this produce?
This is for polymorphic serialization. It will write the class name to the field `@class` (set by property = "@class"). When deserializing an object, it needs the class name to know the concrete class since the reflection APIs will only provide the interface name.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13335
**[Test build #59771 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59771/consoleFull)** for PR 13335 at commit [`8dfc105`](https://github.com/apache/spark/commit/8dfc10524bc3407ff640c9fd1d03d3d13f1c6e50).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13335
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r64955681
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/util/ContinuousQueryListener.scala ---
@@ -65,11 +65,23 @@ object ContinuousQueryListener {
trait Event
/** Event representing the start of a query */
- class QueryStarted private[sql](val query: ContinuousQuery) extends Event
+ class QueryStarted private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/** Event representing any progress updates in a query */
- class QueryProgress private[sql](val query: ContinuousQuery) extends Event
+ class QueryProgress private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/** Event representing that termination of a query */
- class QueryTerminated private[sql](val query: ContinuousQuery) extends Event
+ class QueryTerminated private[sql](val queryInfo: ContinuousQueryInfo) extends Event
}
+
+/**
+ * :: Experimental ::
+ * A class that contains information about [[ContinuousQuery]].
+ */
+@Experimental
+case class ContinuousQueryInfo(
+ name: String,
+ isActive: Boolean,
+ sourceStatuses: Seq[SourceStatus],
+ sinkStatus: SinkStatus,
+ exception: Option[ContinuousQueryException])
--- End diff --
Replaced it with String
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r65963035
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/ContinuousQueryListener.scala ---
@@ -70,26 +71,34 @@ abstract class ContinuousQueryListener {
object ContinuousQueryListener {
/**
- * Base type of [[ContinuousQueryListener]] events.
+ * Base type of [[ContinuousQueryListener]] events
* @since 2.0.0
*/
- trait Event
+ trait Event extends SparkListenerEvent
/**
- * Event representing the start of a query.
+ * Event representing the start of a query
* @since 2.0.0
*/
- class QueryStarted private[sql](val query: ContinuousQuery) extends Event
+ class QueryStarted private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/**
- * Event representing any progress updates in a query.
+ * Event representing any progress updates in a query
* @since 2.0.0
*/
- class QueryProgress private[sql](val query: ContinuousQuery) extends Event
+ class QueryProgress private[sql](val queryInfo: ContinuousQueryInfo) extends Event
/**
- * Event representing that termination of a query.
+ * Event representing that termination of a query
+ *
+ * @param queryInfo The query info.
+ * @param exception The exception message of the [[ContinuousQuery]] if any. Otherwise, it
+ * will be `None`.
+ * @param stackTrace The stack trace of the exception if any.
* @since 2.0.0
*/
- class QueryTerminated private[sql](val query: ContinuousQuery) extends Event
+ class QueryTerminated private[sql](
+ val queryInfo: ContinuousQueryInfo,
+ val exception: Option[String],
+ val stackTrace: Seq[StackTraceElement]) extends Event
--- End diff --
Isnt it more Java friendly to make this Array instead of Seq? It does not need to be Seq.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to make Contin...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13335
**[Test build #59674 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59674/consoleFull)** for PR 13335 at commit [`3669df2`](https://github.com/apache/spark/commit/3669df28f27082364f4b67f2fef18861fc6ca514).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13335: [SPARK-15580][SQL]Add ContinuousQueryInfo to make...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/13335#discussion_r65962876
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/ContinuousQueryListener.scala ---
@@ -70,26 +71,34 @@ abstract class ContinuousQueryListener {
object ContinuousQueryListener {
/**
- * Base type of [[ContinuousQueryListener]] events.
+ * Base type of [[ContinuousQueryListener]] events
* @since 2.0.0
*/
- trait Event
+ trait Event extends SparkListenerEvent
/**
- * Event representing the start of a query.
+ * Event representing the start of a query
* @since 2.0.0
*/
- class QueryStarted private[sql](val query: ContinuousQuery) extends Event
+ class QueryStarted private[sql](val queryInfo: ContinuousQueryInfo) extends Event
--- End diff --
Shouldnt these also be marked with @Experimental?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15580][SQL]Add ContinuousQueryInfo to m...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13335#issuecomment-222012652
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org