You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by HeartSaVioR <gi...@git.apache.org> on 2018/05/03 02:38:09 UTC
[GitHub] spark pull request #21222: [SPARK-24161][SS] Enable debug package feature on...
GitHub user HeartSaVioR opened a pull request:
https://github.com/apache/spark/pull/21222
[SPARK-24161][SS] Enable debug package feature on structured streaming
## What changes were proposed in this pull request?
Currently, debug package has a implicit class "DebugQuery" which matches Dataset to provide debug features on Dataset class. It doesn't work with structured streaming: it requires query is already started, and the information can be retrieved from StreamingQuery, not Dataset. I guess that's why "explain" had to be placed to StreamingQuery whereas it already exists on Dataset.
This patch adds a new implicit class "DebugStreamQuery" which matches StreamingQuery to provide similar debug features on StreamingQuery class.
## How was this patch tested?
Added relevant unit tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HeartSaVioR/spark SPARK-24161
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21222.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21222
----
commit c1ad1c557e6165455457adb6f148d6d9616548a1
Author: Jungtaek Lim <ka...@...>
Date: 2018-05-03T02:26:48Z
SPARK-24161 Enable debug package feature on structured streaming
* added implicit class which adds debug features for StreamingQuery
* added unit tests for new functionalities
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21222: [SPARK-24161][SS] Enable debug package feature on...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21222#discussion_r205961813
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/debug/package.scala ---
@@ -88,23 +100,70 @@ package object debug {
}
}
+ /**
+ * Get WholeStageCodegenExec subtrees and the codegen in a query plan into one String
+ *
+ * @param query the streaming query for codegen
+ * @return single String containing all WholeStageCodegen subtrees and corresponding codegen
+ */
+ def codegenString(query: StreamingQuery): String = {
+ val msg = unwrapStreamingQueryWrapper(query) match {
+ case w: StreamExecution =>
+ if (w.lastExecution != null) {
+ codegenString(w.lastExecution.executedPlan)
+ } else {
+ "No physical plan. Waiting for data."
+ }
+
+ case _ => "Only supported for StreamExecution."
+ }
+ msg
+ }
+
+ /**
+ * Get WholeStageCodegenExec subtrees and the codegen in a query plan
+ *
+ * @param query the streaming query for codegen
+ * @return Sequence of WholeStageCodegen subtrees and corresponding codegen
+ */
+ def codegenStringSeq(query: StreamingQuery): Seq[(String, String)] = {
+ val planAndCodes = unwrapStreamingQueryWrapper(query) match {
+ case w: StreamExecution if w.lastExecution != null =>
+ codegenStringSeq(w.lastExecution.executedPlan)
+
+ case _ => Seq.empty
+ }
+ planAndCodes
+ }
+
+ /* Helper function to reuse duplicated code block between batch and streaming. */
+ private def debugInternal(plan: SparkPlan): Unit = {
+ val visited = new collection.mutable.HashSet[TreeNodeRef]()
+ val debugPlan = plan transform {
+ case s: SparkPlan if !visited.contains(new TreeNodeRef(s)) =>
+ visited += new TreeNodeRef(s)
+ DebugExec(s)
+ }
+ debugPrint(s"Results returned: ${debugPlan.execute().count()}")
+ debugPlan.foreach {
+ case d: DebugExec => d.dumpStats()
+ case _ =>
+ }
+ }
+
+ private def unwrapStreamingQueryWrapper(query: StreamingQuery): StreamingQuery = {
+ query match {
+ case wrapper: StreamingQueryWrapper => wrapper.streamingQuery
+ case _ => query
--- End diff --
OK. I can apply your suggestion but not 100% sure about intention: propagate exception to the caller side (user code), or catch exception inside debug package and handle it.
I'll apply latter case but please let me know when you intended former case.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21222
**[Test build #93744 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93744/testReport)** for PR 21222 at commit [`6f32240`](https://github.com/apache/spark/commit/6f32240e01cbdf6c1ea9ed08b73dbdcf0d2eac7e).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21222
@zsxwing OK I also wonder `debug` is applicable for streaming but wanted to fill the gap earlier. Will remove `debug` for streaming. Will update shortly. Thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21222: [SPARK-24161][SS] Enable debug package feature on...
Posted by attilapiros <gi...@git.apache.org>.
Github user attilapiros commented on a diff in the pull request:
https://github.com/apache/spark/pull/21222#discussion_r207857933
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala ---
@@ -513,6 +515,125 @@ class StreamSuite extends StreamTest {
}
}
+ test("explain-continuous") {
+ val inputData = ContinuousMemoryStream[Int]
+ val df = inputData.toDS().map(_ * 2).filter(_ > 5)
+
+ // Test `df.explain`
+ val explain = ExplainCommand(df.queryExecution.logical, extended = false)
+ val explainString =
+ spark.sessionState
+ .executePlan(explain)
+ .executedPlan
+ .executeCollect()
+ .map(_.getString(0))
+ .mkString("\n")
+ assert(explainString.contains("Filter"))
+ assert(explainString.contains("MapElements"))
+ assert(!explainString.contains("LocalTableScan"))
+
+ // Test StreamingQuery.display
+ val q = df.writeStream.queryName("memory_continuous_explain")
+ .outputMode(OutputMode.Update()).format("memory")
+ .trigger(Trigger.Continuous("1 seconds"))
+ .start()
+ .asInstanceOf[StreamingQueryWrapper]
+ .streamingQuery
+ try {
+ // in continuous mode, the query will be run even there's no data
+ // sleep a bit to ensure initialization
+ eventually(timeout(2.seconds), interval(100.milliseconds)) {
+ assert(q.lastExecution != null)
+ }
+
+ val explainWithoutExtended = q.explainInternal(false)
+
+ // `extended = false` only displays the physical plan.
+ assert("Streaming RelationV2 ContinuousMemoryStream".r
+ .findAllMatchIn(explainWithoutExtended).size === 0)
+ assert("ScanV2 ContinuousMemoryStream".r
+ .findAllMatchIn(explainWithoutExtended).size === 1)
+
+ val explainWithExtended = q.explainInternal(true)
+ // `extended = true` displays 3 logical plans (Parsed/Optimized/Optimized) and 1 physical
+ // plan.
+ assert("Streaming RelationV2 ContinuousMemoryStream".r
+ .findAllMatchIn(explainWithExtended).size === 3)
+ assert("ScanV2 ContinuousMemoryStream".r
+ .findAllMatchIn(explainWithExtended).size === 1)
+ } finally {
+ q.stop()
+ }
+ }
+
+ test("codegen-microbatch") {
+ import org.apache.spark.sql.execution.debug._
+
+ val inputData = MemoryStream[Int]
+ val df = inputData.toDS().map(_ * 2).filter(_ > 5)
+
+ // Test StreamingQuery.codegen
+ val q = df.writeStream.queryName("memory_microbatch_codegen")
+ .outputMode(OutputMode.Update)
+ .format("memory")
+ .trigger(Trigger.ProcessingTime("1 seconds"))
+ .start()
+
+ try {
+ assert("No physical plan. Waiting for data." === codegenString(q))
+ assert(codegenStringSeq(q).isEmpty)
+
+ inputData.addData(1, 2, 3, 4, 5)
+ q.processAllAvailable()
+
+ val codegenStr = codegenString(q)
--- End diff --
What about to extract the following 8 lines into a private method as the `codegenString` and `codegenStringSeq` result checking are the same for microbatch and continues executions?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21222: [SPARK-24161][SS] Enable debug package feature on...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/21222#discussion_r205918043
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala ---
@@ -829,6 +955,18 @@ class StreamSuite extends StreamTest {
assert(query.exception.isEmpty)
}
}
+
+ private def waitForLastExecution(q: StreamExecution): Unit = {
--- End diff --
You can use `eventually` to replace this method.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21222: [SPARK-24161][SS] Enable debug package feature on...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/21222#discussion_r206342230
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/debug/package.scala ---
@@ -88,23 +100,72 @@ package object debug {
}
}
+ /**
+ * Get WholeStageCodegenExec subtrees and the codegen in a query plan into one String
+ *
+ * @param query the streaming query for codegen
+ * @return single String containing all WholeStageCodegen subtrees and corresponding codegen
+ */
+ def codegenString(query: StreamingQuery): String = {
+ try {
+ val w = asStreamExecution(query)
+ if (w.lastExecution != null) {
+ codegenString(w.lastExecution.executedPlan)
+ } else {
+ "No physical plan. Waiting for data."
+ }
+ } catch {
--- End diff --
ditto
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/21222
Thanks! Merging to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21222: [SPARK-24161][SS] Enable debug package feature on...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/21222#discussion_r205917174
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/debug/package.scala ---
@@ -88,23 +100,70 @@ package object debug {
}
}
+ /**
+ * Get WholeStageCodegenExec subtrees and the codegen in a query plan into one String
+ *
+ * @param query the streaming query for codegen
+ * @return single String containing all WholeStageCodegen subtrees and corresponding codegen
+ */
+ def codegenString(query: StreamingQuery): String = {
+ val msg = unwrapStreamingQueryWrapper(query) match {
+ case w: StreamExecution =>
+ if (w.lastExecution != null) {
+ codegenString(w.lastExecution.executedPlan)
+ } else {
+ "No physical plan. Waiting for data."
+ }
+
+ case _ => "Only supported for StreamExecution."
+ }
+ msg
+ }
+
+ /**
+ * Get WholeStageCodegenExec subtrees and the codegen in a query plan
+ *
+ * @param query the streaming query for codegen
+ * @return Sequence of WholeStageCodegen subtrees and corresponding codegen
+ */
+ def codegenStringSeq(query: StreamingQuery): Seq[(String, String)] = {
+ val planAndCodes = unwrapStreamingQueryWrapper(query) match {
+ case w: StreamExecution if w.lastExecution != null =>
+ codegenStringSeq(w.lastExecution.executedPlan)
+
+ case _ => Seq.empty
+ }
+ planAndCodes
+ }
+
+ /* Helper function to reuse duplicated code block between batch and streaming. */
--- End diff --
nit: I prefer to explain what the method does, such as, `print debug information for a plan`. In addition, we usually use `/** ... */` for a method comment.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21222: [SPARK-24161][SS] Enable debug package feature on...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21222#discussion_r205961761
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala ---
@@ -829,6 +955,18 @@ class StreamSuite extends StreamTest {
assert(query.exception.isEmpty)
}
}
+
+ private def waitForLastExecution(q: StreamExecution): Unit = {
--- End diff --
Thanks for letting me know! Addressed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21222
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90240/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21222
**[Test build #93741 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93741/testReport)** for PR 21222 at commit [`6f32240`](https://github.com/apache/spark/commit/6f32240e01cbdf6c1ea9ed08b73dbdcf0d2eac7e).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21222
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93744/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21222
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94309/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21222
**[Test build #90240 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90240/testReport)** for PR 21222 at commit [`b9bad1a`](https://github.com/apache/spark/commit/b9bad1ae6618606e91df35f854025bc32c8178de).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21222: [SPARK-24161][SS] Enable debug package feature on...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21222#discussion_r186247474
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/debug/package.scala ---
@@ -116,6 +168,30 @@ package object debug {
}
}
+ implicit class DebugStreamQuery(query: StreamingQuery) extends Logging {
+ def debug(): Unit = {
+ query match {
+ case w: StreamExecution =>
--- End diff --
My bad. Fixed. Also changed the unit tests so that your reported case can be covered in test.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21222
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90168/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21222
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21222
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21222: [SPARK-24161][SS] Enable debug package feature on...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/21222
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21222
**[Test build #91610 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91610/testReport)** for PR 21222 at commit [`b9bad1a`](https://github.com/apache/spark/commit/b9bad1ae6618606e91df35f854025bc32c8178de).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21222
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94285/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21222
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93741/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21222: [SPARK-24161][SS] Enable debug package feature on...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21222#discussion_r186032252
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/debug/package.scala ---
@@ -88,23 +100,62 @@ package object debug {
}
}
+ /**
+ * Get WholeStageCodegenExec subtrees and the codegen in a query plan into one String
+ *
+ * @param query the streaming query for codegen
+ * @return single String containing all WholeStageCodegen subtrees and corresponding codegen
+ */
+ def codegenString(query: StreamingQuery): String = {
+ val msg = query match {
+ case w: StreamExecution if w.lastExecution != null =>
--- End diff --
Addressed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21222
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21222
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by jose-torres <gi...@git.apache.org>.
Github user jose-torres commented on the issue:
https://github.com/apache/spark/pull/21222
LGTM, but I'm not familiar with this debug package so I can't evaluate if it's being used correctly.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21222
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21222
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94077/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21222
**[Test build #93741 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93741/testReport)** for PR 21222 at commit [`6f32240`](https://github.com/apache/spark/commit/6f32240e01cbdf6c1ea9ed08b73dbdcf0d2eac7e).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21222
**[Test build #90240 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90240/testReport)** for PR 21222 at commit [`b9bad1a`](https://github.com/apache/spark/commit/b9bad1ae6618606e91df35f854025bc32c8178de).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21222
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21222
**[Test build #93744 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93744/testReport)** for PR 21222 at commit [`6f32240`](https://github.com/apache/spark/commit/6f32240e01cbdf6c1ea9ed08b73dbdcf0d2eac7e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21222
**[Test build #94077 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94077/testReport)** for PR 21222 at commit [`80245ba`](https://github.com/apache/spark/commit/80245ba60366f51290127ab5086859d472f7397d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21222
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21222
**[Test build #90168 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90168/testReport)** for PR 21222 at commit [`c1ad1c5`](https://github.com/apache/spark/commit/c1ad1c557e6165455457adb6f148d6d9616548a1).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21222: [SPARK-24161][SS] Enable debug package feature on...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/21222#discussion_r207392732
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/debug/package.scala ---
@@ -116,6 +177,30 @@ package object debug {
}
}
+ implicit class DebugStreamQuery(query: StreamingQuery) extends Logging {
+ def debug(): Unit = {
--- End diff --
Now I see why this doesn't work in continuous mode. It requires to execute `executedPlan`. This may not work in micro batch mode either. Both micro batch mode and continuous mode are not designed to run an internal plan multiple times. E.g., there is a race condition that `debug` may try to run a batch that has been cleaned up. I suggest to remove this method as the result may be confusing.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21222
Thanks @zsxwing for reviewing! Addressed review comments. Please take a look at again. Thanks in advance!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21222: [SPARK-24161][SS] Enable debug package feature on...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/21222#discussion_r206342173
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/debug/package.scala ---
@@ -116,6 +177,30 @@ package object debug {
}
}
+ implicit class DebugStreamQuery(query: StreamingQuery) extends Logging {
+ def debug(): Unit = {
+ try {
+ val w = asStreamExecution(query)
+ if (w.lastExecution == null) {
+ debugPrint("No physical plan. Waiting for data.")
+ } else {
+ val executedPlan = w.lastExecution.executedPlan
+ if (executedPlan.find(_.isInstanceOf[WriteToContinuousDataSourceExec]).isDefined) {
+ debugPrint("Debug on continuous mode is not supported.")
+ } else {
+ debugInternal(executedPlan)
+ }
+ }
+ } catch {
--- End diff --
I prefer to remove this. There is nothing to do in this case. This basically means a bug in Spark: a new type is added but `asStreamExecution` is not updated accordingly.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21222
@zsxwing Kindly reminder.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21222
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93110/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21222
**[Test build #90111 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90111/testReport)** for PR 21222 at commit [`c1ad1c5`](https://github.com/apache/spark/commit/c1ad1c557e6165455457adb6f148d6d9616548a1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` implicit class DebugStreamQuery(query: StreamingQuery) extends Logging `
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21222
retest this, please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21222
@tdas @jose-torres @jerryshao @arunmahadevan
Kindly ping to review.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21222: [SPARK-24161][SS] Enable debug package feature on...
Posted by arunmahadevan <gi...@git.apache.org>.
Github user arunmahadevan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21222#discussion_r186238156
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/debug/package.scala ---
@@ -116,6 +168,30 @@ package object debug {
}
}
+ implicit class DebugStreamQuery(query: StreamingQuery) extends Logging {
+ def debug(): Unit = {
+ query match {
+ case w: StreamExecution =>
--- End diff --
The expectations is that `query.debug` and `query.debugCodegen` should work correct ? I think you need to match `StreamingQuery` and extract the `StreamExecution` here.
At-least in the spark-shell the return value for `DataStreamWriter.start` is a StreamingQueryWrapper.
```
scala> import org.apache.spark.sql.execution.debug._
import org.apache.spark.sql.execution.debug._
scala> val query = windowedCounts.writeStream.outputMode("complete").format("console").option("truncate", "false").start()
query: org.apache.spark.sql.streaming.StreamingQuery = org.apache.spark.sql.execution.streaming.StreamingQueryWrapper@6eb825e4
scala> query.debug
Only supported for StreamExecution.
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21222
ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21222: [SPARK-24161][SS] Enable debug package feature on...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21222#discussion_r207881161
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala ---
@@ -513,6 +515,125 @@ class StreamSuite extends StreamTest {
}
}
+ test("explain-continuous") {
+ val inputData = ContinuousMemoryStream[Int]
+ val df = inputData.toDS().map(_ * 2).filter(_ > 5)
+
+ // Test `df.explain`
+ val explain = ExplainCommand(df.queryExecution.logical, extended = false)
+ val explainString =
+ spark.sessionState
+ .executePlan(explain)
+ .executedPlan
+ .executeCollect()
+ .map(_.getString(0))
+ .mkString("\n")
+ assert(explainString.contains("Filter"))
+ assert(explainString.contains("MapElements"))
+ assert(!explainString.contains("LocalTableScan"))
+
+ // Test StreamingQuery.display
+ val q = df.writeStream.queryName("memory_continuous_explain")
+ .outputMode(OutputMode.Update()).format("memory")
+ .trigger(Trigger.Continuous("1 seconds"))
+ .start()
+ .asInstanceOf[StreamingQueryWrapper]
+ .streamingQuery
+ try {
+ // in continuous mode, the query will be run even there's no data
+ // sleep a bit to ensure initialization
+ eventually(timeout(2.seconds), interval(100.milliseconds)) {
+ assert(q.lastExecution != null)
+ }
+
+ val explainWithoutExtended = q.explainInternal(false)
+
+ // `extended = false` only displays the physical plan.
+ assert("Streaming RelationV2 ContinuousMemoryStream".r
+ .findAllMatchIn(explainWithoutExtended).size === 0)
+ assert("ScanV2 ContinuousMemoryStream".r
+ .findAllMatchIn(explainWithoutExtended).size === 1)
+
+ val explainWithExtended = q.explainInternal(true)
+ // `extended = true` displays 3 logical plans (Parsed/Optimized/Optimized) and 1 physical
+ // plan.
+ assert("Streaming RelationV2 ContinuousMemoryStream".r
+ .findAllMatchIn(explainWithExtended).size === 3)
+ assert("ScanV2 ContinuousMemoryStream".r
+ .findAllMatchIn(explainWithExtended).size === 1)
+ } finally {
+ q.stop()
+ }
+ }
+
+ test("codegen-microbatch") {
+ import org.apache.spark.sql.execution.debug._
+
+ val inputData = MemoryStream[Int]
+ val df = inputData.toDS().map(_ * 2).filter(_ > 5)
+
+ // Test StreamingQuery.codegen
+ val q = df.writeStream.queryName("memory_microbatch_codegen")
+ .outputMode(OutputMode.Update)
+ .format("memory")
+ .trigger(Trigger.ProcessingTime("1 seconds"))
+ .start()
+
+ try {
+ assert("No physical plan. Waiting for data." === codegenString(q))
+ assert(codegenStringSeq(q).isEmpty)
+
+ inputData.addData(1, 2, 3, 4, 5)
+ q.processAllAvailable()
+
+ val codegenStr = codegenString(q)
--- End diff --
Nice finding. Will address.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21222
@zsxwing Addressed review comments. Please take a look again.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21222: [SPARK-24161][SS] Enable debug package feature on...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/21222#discussion_r205917365
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/debug/package.scala ---
@@ -88,23 +100,70 @@ package object debug {
}
}
+ /**
+ * Get WholeStageCodegenExec subtrees and the codegen in a query plan into one String
+ *
+ * @param query the streaming query for codegen
+ * @return single String containing all WholeStageCodegen subtrees and corresponding codegen
+ */
+ def codegenString(query: StreamingQuery): String = {
+ val msg = unwrapStreamingQueryWrapper(query) match {
+ case w: StreamExecution =>
+ if (w.lastExecution != null) {
+ codegenString(w.lastExecution.executedPlan)
+ } else {
+ "No physical plan. Waiting for data."
+ }
+
+ case _ => "Only supported for StreamExecution."
+ }
+ msg
+ }
+
+ /**
+ * Get WholeStageCodegenExec subtrees and the codegen in a query plan
+ *
+ * @param query the streaming query for codegen
+ * @return Sequence of WholeStageCodegen subtrees and corresponding codegen
+ */
+ def codegenStringSeq(query: StreamingQuery): Seq[(String, String)] = {
+ val planAndCodes = unwrapStreamingQueryWrapper(query) match {
+ case w: StreamExecution if w.lastExecution != null =>
+ codegenStringSeq(w.lastExecution.executedPlan)
+
+ case _ => Seq.empty
+ }
+ planAndCodes
+ }
+
+ /* Helper function to reuse duplicated code block between batch and streaming. */
+ private def debugInternal(plan: SparkPlan): Unit = {
+ val visited = new collection.mutable.HashSet[TreeNodeRef]()
+ val debugPlan = plan transform {
+ case s: SparkPlan if !visited.contains(new TreeNodeRef(s)) =>
+ visited += new TreeNodeRef(s)
+ DebugExec(s)
+ }
+ debugPrint(s"Results returned: ${debugPlan.execute().count()}")
+ debugPlan.foreach {
+ case d: DebugExec => d.dumpStats()
+ case _ =>
+ }
+ }
+
+ private def unwrapStreamingQueryWrapper(query: StreamingQuery): StreamingQuery = {
+ query match {
+ case wrapper: StreamingQueryWrapper => wrapper.streamingQuery
+ case _ => query
--- End diff --
I prefer to throw an exception here and make this method return `StreamExecution`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21222
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21222
**[Test build #94285 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94285/testReport)** for PR 21222 at commit [`34a1e5e`](https://github.com/apache/spark/commit/34a1e5e5c5bc4db2447a5db0f63a84e8062ca736).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21222
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21222
**[Test build #91610 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91610/testReport)** for PR 21222 at commit [`b9bad1a`](https://github.com/apache/spark/commit/b9bad1ae6618606e91df35f854025bc32c8178de).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21222
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93071/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21222
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21222
add to whitelist
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21222
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21222: [SPARK-24161][SS] Enable debug package feature on...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/21222#discussion_r205918179
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala ---
@@ -513,6 +514,131 @@ class StreamSuite extends StreamTest {
}
}
+ test("explain-continuous") {
+ val inputData = ContinuousMemoryStream[Int]
+ val df = inputData.toDS().map(_ * 2).filter(_ > 5)
+
+ // Test `df.explain`
+ val explain = ExplainCommand(df.queryExecution.logical, extended = false)
+ val explainString =
+ spark.sessionState
+ .executePlan(explain)
+ .executedPlan
+ .executeCollect()
+ .map(_.getString(0))
+ .mkString("\n")
+ assert(explainString.contains("Filter"))
+ assert(explainString.contains("MapElements"))
+ assert(!explainString.contains("LocalTableScan"))
+
+ // Test StreamingQuery.display
+ val q = df.writeStream.queryName("memory_continuous_explain")
+ .outputMode(OutputMode.Update()).format("memory")
+ .trigger(Trigger.Continuous("1 seconds"))
+ .start()
+ .asInstanceOf[StreamingQueryWrapper]
+ .streamingQuery
+ try {
+ // in continuous mode, the query will be run even there's no data
+ // sleep a bit to ensure initialization
+ waitForLastExecution(q)
+
+ val explainWithoutExtended = q.explainInternal(false)
+
+ print(explainWithoutExtended)
--- End diff --
nit: remove this line
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21222
**[Test build #93110 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93110/testReport)** for PR 21222 at commit [`b9bad1a`](https://github.com/apache/spark/commit/b9bad1ae6618606e91df35f854025bc32c8178de).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21222: [SPARK-24161][SS] Enable debug package feature on...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21222#discussion_r205961764
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala ---
@@ -513,6 +514,131 @@ class StreamSuite extends StreamTest {
}
}
+ test("explain-continuous") {
+ val inputData = ContinuousMemoryStream[Int]
+ val df = inputData.toDS().map(_ * 2).filter(_ > 5)
+
+ // Test `df.explain`
+ val explain = ExplainCommand(df.queryExecution.logical, extended = false)
+ val explainString =
+ spark.sessionState
+ .executePlan(explain)
+ .executedPlan
+ .executeCollect()
+ .map(_.getString(0))
+ .mkString("\n")
+ assert(explainString.contains("Filter"))
+ assert(explainString.contains("MapElements"))
+ assert(!explainString.contains("LocalTableScan"))
+
+ // Test StreamingQuery.display
+ val q = df.writeStream.queryName("memory_continuous_explain")
+ .outputMode(OutputMode.Update()).format("memory")
+ .trigger(Trigger.Continuous("1 seconds"))
+ .start()
+ .asInstanceOf[StreamingQueryWrapper]
+ .streamingQuery
+ try {
+ // in continuous mode, the query will be run even there's no data
+ // sleep a bit to ensure initialization
+ waitForLastExecution(q)
+
+ val explainWithoutExtended = q.explainInternal(false)
+
+ print(explainWithoutExtended)
--- End diff --
Will fix.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21222
Thanks @zsxwing for merging and thanks all for reviewing!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21222
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90111/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21222
**[Test build #94309 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94309/testReport)** for PR 21222 at commit [`34a1e5e`](https://github.com/apache/spark/commit/34a1e5e5c5bc4db2447a5db0f63a84e8062ca736).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/21222
LGTM pending tests
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21222: [SPARK-24161][SS] Enable debug package feature on...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/21222#discussion_r206342211
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/debug/package.scala ---
@@ -88,23 +100,72 @@ package object debug {
}
}
+ /**
+ * Get WholeStageCodegenExec subtrees and the codegen in a query plan into one String
+ *
+ * @param query the streaming query for codegen
+ * @return single String containing all WholeStageCodegen subtrees and corresponding codegen
+ */
+ def codegenString(query: StreamingQuery): String = {
+ try {
+ val w = asStreamExecution(query)
+ if (w.lastExecution != null) {
+ codegenString(w.lastExecution.executedPlan)
+ } else {
+ "No physical plan. Waiting for data."
+ }
+ } catch {
+ case _: IllegalArgumentException => "Only supported for StreamExecution."
+ }
+ }
+
+ /**
+ * Get WholeStageCodegenExec subtrees and the codegen in a query plan
+ *
+ * @param query the streaming query for codegen
+ * @return Sequence of WholeStageCodegen subtrees and corresponding codegen
+ */
+ def codegenStringSeq(query: StreamingQuery): Seq[(String, String)] = {
+ try {
+ val w = asStreamExecution(query)
+ if (w.lastExecution != null) {
+ codegenStringSeq(w.lastExecution.executedPlan)
+ } else {
+ Seq.empty
+ }
+ } catch {
--- End diff --
ditto
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21222
retest this, please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21222
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21222
Kindly ping again to @tdas
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21222
**[Test build #94309 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94309/testReport)** for PR 21222 at commit [`34a1e5e`](https://github.com/apache/spark/commit/34a1e5e5c5bc4db2447a5db0f63a84e8062ca736).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/21222
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21222: [SPARK-24161][SS] Enable debug package feature on...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21222#discussion_r205961782
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/debug/package.scala ---
@@ -116,6 +175,30 @@ package object debug {
}
}
+ implicit class DebugStreamQuery(query: StreamingQuery) extends Logging {
+ def debug(): Unit = {
+ unwrapStreamingQueryWrapper(query) match {
+ case w: StreamExecution =>
+ if (w.lastExecution == null) {
+ debugPrint("No physical plan. Waiting for data.")
+ } else {
+ val executedPlan = w.lastExecution.executedPlan
+ if (executedPlan.find(_.isInstanceOf[WriteToContinuousDataSourceExec]).isDefined) {
+ debugPrint("Debug on continuous mode is not supported.")
--- End diff --
`debug()` will run query shortly to collect information: for micro-batch it will be 1 batch but in continuous mode it will never be finished.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21222
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90186/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21222: [SPARK-24161][SS] Enable debug package feature on...
Posted by arunmahadevan <gi...@git.apache.org>.
Github user arunmahadevan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21222#discussion_r185981329
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/debug/package.scala ---
@@ -88,23 +100,62 @@ package object debug {
}
}
+ /**
+ * Get WholeStageCodegenExec subtrees and the codegen in a query plan into one String
+ *
+ * @param query the streaming query for codegen
+ * @return single String containing all WholeStageCodegen subtrees and corresponding codegen
+ */
+ def codegenString(query: StreamingQuery): String = {
+ val msg = query match {
+ case w: StreamExecution if w.lastExecution != null =>
--- End diff --
I think this can be rewritten as if `case w: StreamExecution => if (w.lastExecution != null) ... else ...`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21222
**[Test build #94285 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94285/testReport)** for PR 21222 at commit [`34a1e5e`](https://github.com/apache/spark/commit/34a1e5e5c5bc4db2447a5db0f63a84e8062ca736).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21222
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21222
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21222
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21222: [SPARK-24161][SS] Enable debug package feature on...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/21222#discussion_r205916804
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/debug/package.scala ---
@@ -116,6 +175,30 @@ package object debug {
}
}
+ implicit class DebugStreamQuery(query: StreamingQuery) extends Logging {
+ def debug(): Unit = {
+ unwrapStreamingQueryWrapper(query) match {
+ case w: StreamExecution =>
+ if (w.lastExecution == null) {
+ debugPrint("No physical plan. Waiting for data.")
+ } else {
+ val executedPlan = w.lastExecution.executedPlan
+ if (executedPlan.find(_.isInstanceOf[WriteToContinuousDataSourceExec]).isDefined) {
+ debugPrint("Debug on continuous mode is not supported.")
--- End diff --
What's the issue you hit here?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21222
**[Test build #94077 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94077/testReport)** for PR 21222 at commit [`80245ba`](https://github.com/apache/spark/commit/80245ba60366f51290127ab5086859d472f7397d).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21222
ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21222
**[Test build #93071 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93071/testReport)** for PR 21222 at commit [`b9bad1a`](https://github.com/apache/spark/commit/b9bad1ae6618606e91df35f854025bc32c8178de).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21222
**[Test build #90111 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90111/testReport)** for PR 21222 at commit [`c1ad1c5`](https://github.com/apache/spark/commit/c1ad1c557e6165455457adb6f148d6d9616548a1).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21222
Kindly ping. I guess debugging last batch might not be attractive that much, but printing codegen would be helpful to someone who want to investigate or debug in detail.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21222
**[Test build #90186 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90186/testReport)** for PR 21222 at commit [`8322bff`](https://github.com/apache/spark/commit/8322bff226bf2080eb4944f26dba9b8dd50cd639).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21222: [SPARK-24161][SS] Enable debug package feature on...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21222#discussion_r205961840
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/debug/package.scala ---
@@ -88,23 +100,70 @@ package object debug {
}
}
+ /**
+ * Get WholeStageCodegenExec subtrees and the codegen in a query plan into one String
+ *
+ * @param query the streaming query for codegen
+ * @return single String containing all WholeStageCodegen subtrees and corresponding codegen
+ */
+ def codegenString(query: StreamingQuery): String = {
+ val msg = unwrapStreamingQueryWrapper(query) match {
+ case w: StreamExecution =>
+ if (w.lastExecution != null) {
+ codegenString(w.lastExecution.executedPlan)
+ } else {
+ "No physical plan. Waiting for data."
+ }
+
+ case _ => "Only supported for StreamExecution."
+ }
+ msg
+ }
+
+ /**
+ * Get WholeStageCodegenExec subtrees and the codegen in a query plan
+ *
+ * @param query the streaming query for codegen
+ * @return Sequence of WholeStageCodegen subtrees and corresponding codegen
+ */
+ def codegenStringSeq(query: StreamingQuery): Seq[(String, String)] = {
+ val planAndCodes = unwrapStreamingQueryWrapper(query) match {
+ case w: StreamExecution if w.lastExecution != null =>
+ codegenStringSeq(w.lastExecution.executedPlan)
+
+ case _ => Seq.empty
+ }
+ planAndCodes
+ }
+
+ /* Helper function to reuse duplicated code block between batch and streaming. */
--- End diff --
Will address.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21222
**[Test build #90186 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90186/testReport)** for PR 21222 at commit [`8322bff`](https://github.com/apache/spark/commit/8322bff226bf2080eb4944f26dba9b8dd50cd639).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21222
adding cc. to @zsxwing since he has been reviewing PRs for SS so far.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21222
**[Test build #93071 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93071/testReport)** for PR 21222 at commit [`b9bad1a`](https://github.com/apache/spark/commit/b9bad1ae6618606e91df35f854025bc32c8178de).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21222
**[Test build #93110 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93110/testReport)** for PR 21222 at commit [`b9bad1a`](https://github.com/apache/spark/commit/b9bad1ae6618606e91df35f854025bc32c8178de).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21222
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91610/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21222
**[Test build #90168 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90168/testReport)** for PR 21222 at commit [`c1ad1c5`](https://github.com/apache/spark/commit/c1ad1c557e6165455457adb6f148d6d9616548a1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` implicit class DebugStreamQuery(query: StreamingQuery) extends Logging `
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21222
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org