You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/05/15 15:10:38 UTC
[GitHub] [spark] ivoson opened a new pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
ivoson opened a new pull request #32557:
URL: https://github.com/apache/spark/pull/32557
### What changes were proposed in this pull request?
Write out Seq of product objects which contain TreeNode, to avoid the cases as described in https://issues.apache.org/jira/browse/SPARK-35411 that essential information will be ignored and just written out as null values. These information are necessary to understand the query plans.
### Why are the changes needed?
Information like cteRelations in With node, and branches in CaseWhen expression are necessary to understand the query plans, they should be written out to the result json string.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
UT case added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wezhang commented on a change in pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
wezhang commented on a change in pull request #32557:
URL: https://github.com/apache/spark/pull/32557#discussion_r633981777
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -997,9 +997,11 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product with Tre
("deserialized" -> s.deserialized) ~ ("replication" -> s.replication)
case n: TreeNode[_] => n.jsonValue
case o: Option[_] => o.map(parseToJson)
- // Recursive scan Seq[TreeNode], Seq[Partitioning], Seq[DataType]
+ // Recursive scan Seq[TreeNode], Seq[Partitioning], Seq[DataType], Seq[Product]
case t: Seq[_] if t.forall(_.isInstanceOf[TreeNode[_]]) ||
- t.forall(_.isInstanceOf[Partitioning]) || t.forall(_.isInstanceOf[DataType]) =>
+ t.forall(_.isInstanceOf[Partitioning]) ||
+ t.forall(_.isInstanceOf[DataType]) ||
+ t.forall(_.isInstanceOf[Product]) =>
Review comment:
The `TreeNode` inherits `Product`, I'm wondering if only one check should be kept.
Maybe this change is redundant, the following one which changes `case p if p.productIterator.exists(_.isInstanceOf[TreeNode[_]]) => true` is the key one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ivoson commented on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
ivoson commented on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-841975036
cc @cloud-fan could you help review this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-842802198
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43179/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-842802198
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43179/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-842896227
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138658/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-843064452
**[Test build #138675 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138675/testReport)** for PR 32557 at commit [`c307dd0`](https://github.com/apache/spark/commit/c307dd0c0c65ceefec69c9612ede4e8ee7670515).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-843101274
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43196/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32557:
URL: https://github.com/apache/spark/pull/32557#discussion_r633602491
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -1001,6 +1001,9 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product with Tre
case t: Seq[_] if t.forall(_.isInstanceOf[TreeNode[_]]) ||
t.forall(_.isInstanceOf[Partitioning]) || t.forall(_.isInstanceOf[DataType]) =>
JArray(t.map(parseToJson).toList)
+ // Scan Seq[Product] to make sure essential information can be written out.
+ case t: Seq[_] if t.forall(_.isInstanceOf[Product]) =>
Review comment:
shall we merge it into the previous case? `case t: Seq[_] if t.forall(_.isInstanceOf[TreeNode[_]]) || ...|| t.forall(_.isInstanceOf[Product])`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-843139789
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43196/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-843283641
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138675/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #32557:
URL: https://github.com/apache/spark/pull/32557
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-842391347
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-842760405
**[Test build #138658 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138658/testReport)** for PR 32557 at commit [`902893c`](https://github.com/apache/spark/commit/902893c6be807ab15373f80f654480d2526a69f1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-843139789
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43196/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-843283641
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138675/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ivoson commented on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
ivoson commented on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-841975036
cc @cloud-fan could you help review this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-843264457
thanks, merging to master/3.1!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-842875464
**[Test build #138658 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138658/testReport)** for PR 32557 at commit [`902893c`](https://github.com/apache/spark/commit/902893c6be807ab15373f80f654480d2526a69f1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-841680826
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-842424093
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32557:
URL: https://github.com/apache/spark/pull/32557#discussion_r633602491
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -1001,6 +1001,9 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product with Tre
case t: Seq[_] if t.forall(_.isInstanceOf[TreeNode[_]]) ||
t.forall(_.isInstanceOf[Partitioning]) || t.forall(_.isInstanceOf[DataType]) =>
JArray(t.map(parseToJson).toList)
+ // Scan Seq[Product] to make sure essential information can be written out.
+ case t: Seq[_] if t.forall(_.isInstanceOf[Product]) =>
Review comment:
shall we merge it into the previous case? `case t: Seq[_] if t.forall(_.isInstanceOf[TreeNode[_]]) || ...|| t.forall(_.isInstanceOf[Product])`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-842391347
**[Test build #138637 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138637/testReport)** for PR 32557 at commit [`b28d6cf`](https://github.com/apache/spark/commit/b28d6cf40f6b7e31d2ad536b886180dada96e2b1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-842388974
ok to test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-842583083
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138637/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-842760405
**[Test build #138658 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138658/testReport)** for PR 32557 at commit [`902893c`](https://github.com/apache/spark/commit/902893c6be807ab15373f80f654480d2526a69f1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ivoson commented on a change in pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
ivoson commented on a change in pull request #32557:
URL: https://github.com/apache/spark/pull/32557#discussion_r634256260
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -997,9 +997,11 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product with Tre
("deserialized" -> s.deserialized) ~ ("replication" -> s.replication)
case n: TreeNode[_] => n.jsonValue
case o: Option[_] => o.map(parseToJson)
- // Recursive scan Seq[TreeNode], Seq[Partitioning], Seq[DataType]
+ // Recursive scan Seq[TreeNode], Seq[Partitioning], Seq[DataType], Seq[Product]
case t: Seq[_] if t.forall(_.isInstanceOf[TreeNode[_]]) ||
- t.forall(_.isInstanceOf[Partitioning]) || t.forall(_.isInstanceOf[DataType]) =>
+ t.forall(_.isInstanceOf[Partitioning]) ||
+ t.forall(_.isInstanceOf[DataType]) ||
+ t.forall(_.isInstanceOf[Product]) =>
Review comment:
Nice catch, I think we can remove the TreeNode check. The code change is for Seq, without this Seq[Product] will be written as null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-843255908
**[Test build #138675 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138675/testReport)** for PR 32557 at commit [`c307dd0`](https://github.com/apache/spark/commit/c307dd0c0c65ceefec69c9612ede4e8ee7670515).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ivoson commented on a change in pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
ivoson commented on a change in pull request #32557:
URL: https://github.com/apache/spark/pull/32557#discussion_r633937423
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -1001,6 +1001,9 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product with Tre
case t: Seq[_] if t.forall(_.isInstanceOf[TreeNode[_]]) ||
t.forall(_.isInstanceOf[Partitioning]) || t.forall(_.isInstanceOf[DataType]) =>
JArray(t.map(parseToJson).toList)
+ // Scan Seq[Product] to make sure essential information can be written out.
+ case t: Seq[_] if t.forall(_.isInstanceOf[Product]) =>
Review comment:
Sure, make sense. Will merge it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ivoson commented on a change in pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
ivoson commented on a change in pull request #32557:
URL: https://github.com/apache/spark/pull/32557#discussion_r633962343
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -1001,6 +1001,9 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product with Tre
case t: Seq[_] if t.forall(_.isInstanceOf[TreeNode[_]]) ||
t.forall(_.isInstanceOf[Partitioning]) || t.forall(_.isInstanceOf[DataType]) =>
JArray(t.map(parseToJson).toList)
+ // Scan Seq[Product] to make sure essential information can be written out.
+ case t: Seq[_] if t.forall(_.isInstanceOf[Product]) =>
Review comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-842779123
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43179/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-843064452
**[Test build #138675 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138675/testReport)** for PR 32557 at commit [`c307dd0`](https://github.com/apache/spark/commit/c307dd0c0c65ceefec69c9612ede4e8ee7670515).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-842566204
**[Test build #138637 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138637/testReport)** for PR 32557 at commit [`b28d6cf`](https://github.com/apache/spark/commit/b28d6cf40f6b7e31d2ad536b886180dada96e2b1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-842388974
ok to test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-841680826
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ivoson commented on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
ivoson commented on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-843270096
Thanks for the review. @cloud-fan @wezhang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-842429047
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43157/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ivoson commented on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
ivoson commented on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-843065222
Gentle ping @cloud-fan , comments addressed, please take another look when you have time. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-841680826
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-843129404
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43196/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-842791856
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43179/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-842583083
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138637/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-842429047
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43157/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-842896227
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138658/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-842391347
**[Test build #138637 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138637/testReport)** for PR 32557 at commit [`b28d6cf`](https://github.com/apache/spark/commit/b28d6cf40f6b7e31d2ad536b886180dada96e2b1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32557: [SPARK-35411][SQL] Add essential information while serializing TreeNode to json
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32557:
URL: https://github.com/apache/spark/pull/32557#issuecomment-842429047
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43157/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org