You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by yaooqinn <gi...@git.apache.org> on 2017/09/27 08:28:34 UTC
[GitHub] spark pull request #19363: [Minor]Override toString of KeyValueGroupedDatase...
GitHub user yaooqinn opened a pull request:
https://github.com/apache/spark/pull/19363
[Minor]Override toString of KeyValueGroupedDataset
## What changes were proposed in this pull request?
#### before
```scala
scala> val words = spark.read.textFile("fREADME.md").flatMap(_.split(" "))
words: org.apache.spark.sql.Dataset[String] = [value: string]
scala> val grouped = words.groupByKey(identity)
grouped: org.apache.spark.sql.KeyValueGroupedDataset[String,String] = org.apache.spark.sql.KeyValueGroupedDataset@65214862
```
#### after
```scala
scala> val words = spark.read.textFile("README.md").flatMap(_.split(" "))
words: org.apache.spark.sql.Dataset[String] = [value: string]
scala> val grouped = words.groupByKey(identity)
grouped: org.apache.spark.sql.KeyValueGroupedDataset[String,String] = [key: [value: string], value: [value: string]]
```
## How was this patch tested?
existing ut
cc @gatorsmile @cloud-fan
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yaooqinn/spark minor-dataset-tostring
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19363.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19363
----
commit a9b30bd6508421de192d0d44b6bc03afd3e0a792
Author: Kent Yao <ya...@hotmail.com>
Date: 2017-09-27T07:54:41Z
override toString of KeyValueGroupedDataset
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][SQL]Override toString of KeyValue/Relation...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19363
**[Test build #82786 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82786/testReport)** for PR 19363 at commit [`1c0a6d0`](https://github.com/apache/spark/commit/1c0a6d0ea4e1e63d9ae51a70178f58d60cf3ad29).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [Minor]Override toString of KeyValueGroupedDataset
Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/19363
cc again @cloud-fan @gatorsmile
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][Minor]Override toString of KeyValue/Relati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19363
**[Test build #82549 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82549/testReport)** for PR 19363 at commit [`f051c10`](https://github.com/apache/spark/commit/f051c1070566b460c98ea49455a158a89dfecdef).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [Minor]Override toString of KeyValueGroupedDataset
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19363
**[Test build #82259 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82259/testReport)** for PR 19363 at commit [`289b23d`](https://github.com/apache/spark/commit/289b23d2536cf190a40f4a5494a57e0854771e10).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19363: [SPARK-22224][Minor]Override toString of KeyValue...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19363#discussion_r143717543
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -564,4 +565,30 @@ class KeyValueGroupedDataset[K, V] private[sql](
encoder: Encoder[R]): Dataset[R] = {
cogroup(other)((key, left, right) => f.call(key, left.asJava, right.asJava).asScala)(encoder)
}
+
+ override def toString: String = {
+ try {
+ val builder = new StringBuilder
+ val kFields = kExprEnc.schema.map {
+ case f => s"${f.name}: ${f.dataType.simpleString(2)}"
+ }
+ val vFields = vExprEnc.schema.map {
+ case f => s"${f.name}: ${f.dataType.simpleString(2)}"
+ }
+ builder.append("[key: [")
+ builder.append(kFields.take(2).mkString(", "))
+ if (kFields.length > 2) {
+ builder.append(" ... " + (kFields.length - 2) + " more field(s)")
+ }
+ builder.append("], value: [")
+ builder.append(vFields.take(2).mkString(", "))
+ if (vFields.length > 2) {
+ builder.append(" ... " + (vFields.length - 2) + " more field(s)")
+ }
+ builder.append("]]").toString()
+ } catch {
+ case NonFatal(e) =>
--- End diff --
same question
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19363: [SPARK-22224][SQL]Override toString of KeyValue/R...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19363#discussion_r144753415
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/KeyValueGroupedDatasetSuite.scala ---
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.spark.sql.test.SharedSQLContext
+
+class KeyValueGroupedDatasetSuite extends QueryTest with SharedSQLContext {
--- End diff --
we can put the tests in `DataSetSuite`, as `DataSet.toString` is also tested there.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19363: [SPARK-22224][Minor]Override toString of KeyValue...
Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/19363#discussion_r143904084
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -564,4 +565,30 @@ class KeyValueGroupedDataset[K, V] private[sql](
encoder: Encoder[R]): Dataset[R] = {
cogroup(other)((key, left, right) => f.call(key, left.asJava, right.asJava).asScala)(encoder)
}
+
+ override def toString: String = {
+ try {
+ val builder = new StringBuilder
+ val kFields = kExprEnc.schema.map {
+ case f => s"${f.name}: ${f.dataType.simpleString(2)}"
+ }
+ val vFields = vExprEnc.schema.map {
+ case f => s"${f.name}: ${f.dataType.simpleString(2)}"
+ }
+ builder.append("[key: [")
+ builder.append(kFields.take(2).mkString(", "))
+ if (kFields.length > 2) {
+ builder.append(" ... " + (kFields.length - 2) + " more field(s)")
+ }
+ builder.append("], value: [")
+ builder.append(vFields.take(2).mkString(", "))
+ if (vFields.length > 2) {
+ builder.append(" ... " + (vFields.length - 2) + " more field(s)")
+ }
+ builder.append("]]").toString()
+ } catch {
+ case NonFatal(e) =>
--- End diff --
I once thought you would answer that for me https://github.com/apache/spark/blame/master/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L374, LOL
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [Minor]Override toString of KeyValueGroupedDataset
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19363
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][Minor]Override toString of KeyValue/Relati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19363
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82574/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][Minor]Override toString of KeyValue/Relati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19363
**[Test build #82574 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82574/testReport)** for PR 19363 at commit [`fe0d64a`](https://github.com/apache/spark/commit/fe0d64a1d5080d10fe6743f725107221acb9dd62).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19363: [SPARK-22224][SQL]Override toString of KeyValue/R...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19363#discussion_r144873850
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/RelationalGroupedDataset.scala ---
@@ -479,7 +492,9 @@ private[sql] object RelationalGroupedDataset {
/**
* The Grouping Type
*/
- private[sql] trait GroupType
+ private[sql] trait GroupType {
+ override def toString: String = getClass.getSimpleName.replace("$", "")
--- End diff --
`getClass.getSimpleName.stripSuffix("$").stripSuffix("Type")`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][SQL]Override toString of KeyValue/Relation...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19363
**[Test build #82828 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82828/testReport)** for PR 19363 at commit [`955e3c9`](https://github.com/apache/spark/commit/955e3c9a560e4b4cda3b73533654949f78cb6ec9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][SQL]Override toString of KeyValue/Relation...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19363
**[Test build #82828 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82828/testReport)** for PR 19363 at commit [`955e3c9`](https://github.com/apache/spark/commit/955e3c9a560e4b4cda3b73533654949f78cb6ec9).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19363: [SPARK-22224][SQL]Override toString of KeyValue/R...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19363#discussion_r144500085
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -564,4 +565,30 @@ class KeyValueGroupedDataset[K, V] private[sql](
encoder: Encoder[R]): Dataset[R] = {
cogroup(other)((key, left, right) => f.call(key, left.asJava, right.asJava).asScala)(encoder)
}
+
+ override def toString: String = {
+ try {
+ val builder = new StringBuilder
+ val kFields = kExprEnc.schema.map {
+ case f => s"${f.name}: ${f.dataType.simpleString(2)}"
+ }
+ val vFields = vExprEnc.schema.map {
+ case f => s"${f.name}: ${f.dataType.simpleString(2)}"
+ }
+ builder.append("[key: [")
+ builder.append(kFields.take(2).mkString(", "))
+ if (kFields.length > 2) {
+ builder.append(" ... " + (kFields.length - 2) + " more field(s)")
+ }
+ builder.append("], value: [")
+ builder.append(vFields.take(2).mkString(", "))
+ if (vFields.length > 2) {
+ builder.append(" ... " + (vFields.length - 2) + " more field(s)")
+ }
+ builder.append("]]").toString()
+ } catch {
+ case NonFatal(e) =>
--- End diff --
Although some methods of `StringBuilder` can throw exceptions, I didn't see simple `append` can throw exception. Access of `schema` should not throw exception at all. So I don't get why it is.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [Minor]Override toString of KeyValueGroupedDataset
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19363
**[Test build #82259 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82259/testReport)** for PR 19363 at commit [`289b23d`](https://github.com/apache/spark/commit/289b23d2536cf190a40f4a5494a57e0854771e10).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][Minor]Override toString of KeyValue/Relati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19363
**[Test build #82619 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82619/testReport)** for PR 19363 at commit [`c02f633`](https://github.com/apache/spark/commit/c02f63390658df953b76c607b894de2abd889e8c).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19363: [Minor]Override toString of KeyValueGroupedDatase...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19363#discussion_r143094589
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -18,16 +18,17 @@
package org.apache.spark.sql
import scala.collection.JavaConverters._
+import scala.util.control.NonFatal
import org.apache.spark.annotation.{Experimental, InterfaceStability}
import org.apache.spark.api.java.function._
import org.apache.spark.sql.catalyst.encoders.{encoderFor, ExpressionEncoder}
import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute, CreateStruct}
import org.apache.spark.sql.catalyst.plans.logical._
-import org.apache.spark.sql.catalyst.streaming.InternalOutputModes
import org.apache.spark.sql.execution.QueryExecution
import org.apache.spark.sql.expressions.ReduceAggregator
import org.apache.spark.sql.streaming.{GroupState, GroupStateTimeout, OutputMode}
+import org.apache.spark.sql.types.StructType
--- End diff --
Why import `StructType`? I didn't see you use it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [Minor]Override toString of KeyValueGroupedDataset
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19363
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82259/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [Minor]Override toString of KeyValueGroupedDataset
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19363
**[Test build #82548 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82548/testReport)** for PR 19363 at commit [`7e7ed19`](https://github.com/apache/spark/commit/7e7ed190c529d1bc14439d0c218640d30a4065e4).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19363: [SPARK-22224][SQL]Override toString of KeyValue/R...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19363#discussion_r144753173
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/RelationalGroupedDataset.scala ---
@@ -465,6 +466,19 @@ class RelationalGroupedDataset protected[sql](
Dataset.ofRows(df.sparkSession, plan)
}
+
+ override def toString: String = {
+ val builder = new StringBuilder
+ builder.append("RelationalGroupedDataset: [groupingBy: [")
+ val kFields = groupingExprs.map(_.asInstanceOf[NamedExpression]).map {
+ case f => s"${f.name}: ${f.dataType.simpleString(2)}"
+ }
+ builder.append(kFields.take(2).mkString(", "))
+ if (kFields.length > 2) {
+ builder.append(" ... " + (kFields.length - 2) + " more field(s)")
+ }
+ builder.append(s"], df: ${df.toString}, type: $groupType]").toString()
--- End diff --
why include `df` here?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [Minor]Override toString of KeyValueGroupedDataset
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19363
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82238/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19363: [SPARK-22224][SQL]Override toString of KeyValue/R...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19363#discussion_r144753249
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -564,4 +564,25 @@ class KeyValueGroupedDataset[K, V] private[sql](
encoder: Encoder[R]): Dataset[R] = {
cogroup(other)((key, left, right) => f.call(key, left.asJava, right.asJava).asScala)(encoder)
}
+
+ override def toString: String = {
+ val builder = new StringBuilder
+ val kFields = kExprEnc.schema.map {
+ case f => s"${f.name}: ${f.dataType.simpleString(2)}"
+ }
+ val vFields = vExprEnc.schema.map {
+ case f => s"${f.name}: ${f.dataType.simpleString(2)}"
+ }
+ builder.append("KeyValueGroupedDataset: [key: [")
+ builder.append(kFields.take(2).mkString(", "))
+ if (kFields.length > 2) {
+ builder.append(" ... " + (kFields.length - 2) + " more field(s)")
+ }
+ builder.append("], value: [")
+ builder.append(vFields.take(2).mkString(", "))
+ if (vFields.length > 2) {
+ builder.append(" ... " + (vFields.length - 2) + " more field(s)")
+ }
+ builder.append("]]").toString()
--- End diff --
shall we include `df` here?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19363: [SPARK-22224][SQL]Override toString of KeyValue/R...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19363#discussion_r144501447
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/RelationalGroupedDataset.scala ---
@@ -465,6 +466,24 @@ class RelationalGroupedDataset protected[sql](
Dataset.ofRows(df.sparkSession, plan)
}
+
+ override def toString: String = {
+ try {
+ val builder = new StringBuilder
+ builder.append("RelationalGroupedDataset: [key: [")
--- End diff --
`key:` or `groupingBy:`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][Minor]Override toString of KeyValue/Relati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19363
**[Test build #82548 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82548/testReport)** for PR 19363 at commit [`7e7ed19`](https://github.com/apache/spark/commit/7e7ed190c529d1bc14439d0c218640d30a4065e4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][Minor]Override toString of KeyValue/Relati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19363
**[Test build #82574 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82574/testReport)** for PR 19363 at commit [`fe0d64a`](https://github.com/apache/spark/commit/fe0d64a1d5080d10fe6743f725107221acb9dd62).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [Minor]Override toString of KeyValueGroupedDataset
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19363
cc @viirya Could you review this PR?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [Minor]Override toString of KeyValueGroupedDataset
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19363
**[Test build #82238 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82238/testReport)** for PR 19363 at commit [`a9b30bd`](https://github.com/apache/spark/commit/a9b30bd6508421de192d0d44b6bc03afd3e0a792).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][Minor]Override toString of KeyValue/Relati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19363
**[Test build #82549 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82549/testReport)** for PR 19363 at commit [`f051c10`](https://github.com/apache/spark/commit/f051c1070566b460c98ea49455a158a89dfecdef).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [Minor]Override toString of KeyValueGroupedDataset
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19363
**[Test build #82238 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82238/testReport)** for PR 19363 at commit [`a9b30bd`](https://github.com/apache/spark/commit/a9b30bd6508421de192d0d44b6bc03afd3e0a792).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][SQL]Override toString of KeyValue/Relation...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19363
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [Minor]Override toString of KeyValueGroupedDataset
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19363
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [Minor]Override toString of KeyValueGroupedDataset
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19363
shall we fix RelationalGroupedDataset too?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [Minor]Override toString of KeyValueGroupedDataset
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19363
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][Minor]Override toString of KeyValue/Relati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19363
**[Test build #82619 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82619/testReport)** for PR 19363 at commit [`c02f633`](https://github.com/apache/spark/commit/c02f63390658df953b76c607b894de2abd889e8c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][Minor]Override toString of KeyValue/Relati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19363
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19363: [Minor]Override toString of KeyValueGroupedDatase...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19363#discussion_r143094398
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -564,4 +565,30 @@ class KeyValueGroupedDataset[K, V] private[sql](
encoder: Encoder[R]): Dataset[R] = {
cogroup(other)((key, left, right) => f.call(key, left.asJava, right.asJava).asScala)(encoder)
}
+
+ override def toString: String = {
+ try {
+ val builder = new StringBuilder
+ val kFields = kExprEnc.schema.map {
+ case f => s"${f.name}: ${f.dataType.simpleString(2)}"
+ }
+ val vFields = vExprEnc.schema.map {
+ case f => s"${f.name}: ${f.dataType.simpleString(2)}"
+ }
+ builder.append("[key: [")
+ builder.append(kFields.take(2).mkString(", "))
+ if (kFields.length > 2) {
+ builder.append(" ... " + (kFields.length - 2) + " more field(s)")
+ }
+ builder.append("], value: [")
+ builder.append(vFields.take(2).mkString(", "))
+ if (vFields.length > 2) {
+ builder.append(" ... " + (vFields.length - 2) + " more field(s)")
+ }
+ builder.append("]]").toString()
+ } catch {
+ case NonFatal(e) =>
--- End diff --
When we will encounter this error?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][SQL]Override toString of KeyValue/Relation...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19363
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82776/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19363: [SPARK-22224][SQL]Override toString of KeyValue/R...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19363#discussion_r144502680
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/RelationalGroupedDatasetSuite.scala ---
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.spark.sql.test.SharedSQLContext
+
+class RelationalGroupedDatasetSuite extends QueryTest with SharedSQLContext {
+ import testImplicits._
+
+ test("Check RelationalGroupedDataset toString: Single data") {
+ val kvDataset = (1 to 3).toDF("id").as[SingleData].groupBy("id")
+ val expected = "RelationalGroupedDataset: [key: [id: int], value: [id: int], GroupByType]"
+ val actual = kvDataset.toString
+ checkString(expected, actual)
+ }
+
+ test("Check RelationalGroupedDataset toString: over length schema ") {
+ val kvDataset = (1 to 3).map( x => (x, x.toString, x.toLong))
+ .toDF("id", "val1", "val2").as[TripleData].groupBy("id")
--- End diff --
Is `TripleData` necessary?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19363: [SPARK-22224][SQL]Override toString of KeyValue/R...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19363#discussion_r144501966
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala ---
@@ -46,6 +36,20 @@ abstract class QueryTest extends PlanTest {
Locale.setDefault(Locale.US)
/**
+ * Makes sure the answer matches the expected result
+ */
+ def checkString(expected: String, actual: String): Unit = {
+ if (expected != actual) {
+ fail(
+ "KeyValueGroupedDataset.toString() gives wrong result:\n\n" + sideBySide(
--- End diff --
You didn't only use this against `KeyValueGroupedDataset`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19363: [SPARK-22224][SQL]Override toString of KeyValue/R...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19363#discussion_r144501685
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/RelationalGroupedDataset.scala ---
@@ -465,6 +466,24 @@ class RelationalGroupedDataset protected[sql](
Dataset.ofRows(df.sparkSession, plan)
}
+
+ override def toString: String = {
+ try {
+ val builder = new StringBuilder
+ builder.append("RelationalGroupedDataset: [key: [")
+ val kFields = groupingExprs.map(_.asInstanceOf[NamedExpression]).map {
+ case f => s"${f.name}: ${f.dataType.simpleString(2)}"
+ }
+ builder.append(kFields.take(2).mkString(", "))
+ if (kFields.length > 2) {
+ builder.append(" ... " + (kFields.length - 2) + " more field(s)")
+ }
+ builder.append(s"], value: ${df.toString}, $groupType]").toString()
--- End diff --
`value:` or `df:`?
`$groupType` -> `type: $groupType`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19363: [SPARK-22224][SQL]Override toString of KeyValue/R...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/19363
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][SQL]Override toString of KeyValue/Relation...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19363
**[Test build #82786 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82786/testReport)** for PR 19363 at commit [`1c0a6d0`](https://github.com/apache/spark/commit/1c0a6d0ea4e1e63d9ae51a70178f58d60cf3ad29).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19363: [SPARK-22224][SQL]Override toString of KeyValue/R...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19363#discussion_r144753446
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/RelationalGroupedDatasetSuite.scala ---
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.spark.sql.test.SharedSQLContext
+
+class RelationalGroupedDatasetSuite extends QueryTest with SharedSQLContext {
--- End diff --
we can put the tests in `DataSetSuite`, as `DataSet.toString` is also tested there.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][Minor]Override toString of KeyValue/Relati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19363
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82619/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [Minor]Override toString of KeyValueGroupedDataset
Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/19363
Ok，i will add a jira tgt and fix RelationalGroupedDataset
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][SQL]Override toString of KeyValue/Relation...
Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/19363
cc @viirya @cloud-fan @gatorsmile
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][SQL]Override toString of KeyValue/Relation...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19363
merging to master!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][SQL]Override toString of KeyValue/Relation...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19363
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [Minor]Override toString of KeyValueGroupedDataset
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/19363
Maybe add a simple test?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19363: [SPARK-22224][SQL]Override toString of KeyValue/R...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19363#discussion_r144500697
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/RelationalGroupedDataset.scala ---
@@ -465,6 +466,24 @@ class RelationalGroupedDataset protected[sql](
Dataset.ofRows(df.sparkSession, plan)
}
+
+ override def toString: String = {
+ try {
+ val builder = new StringBuilder
+ builder.append("RelationalGroupedDataset: [key: [")
+ val kFields = groupingExprs.map(_.asInstanceOf[NamedExpression]).map {
--- End diff --
Are all expressions in `groupingExprs` `NamedExpression`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][Minor]Override toString of KeyValue/Relati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19363
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82549/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][SQL]Override toString of KeyValue/Relation...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19363
**[Test build #82776 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82776/testReport)** for PR 19363 at commit [`9166640`](https://github.com/apache/spark/commit/91666403e287015d953c4b105df6bc35dae966c3).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [Minor]Override toString of KeyValueGroupedDataset
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19363
ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][SQL]Override toString of KeyValue/Relation...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19363
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82786/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19363: [Minor]Override toString of KeyValueGroupedDatase...
Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/19363#discussion_r141509523
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -54,6 +55,14 @@ class KeyValueGroupedDataset[K, V] private[sql](
private def sparkSession = queryExecution.sparkSession
/**
+ * Returns the schema of this Dataset.
+ *
+ * @group basic
+ * @since 2.3.0
+ */
+ def schema: StructType = queryExecution.analyzed.schema
--- End diff --
ok
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][Minor]Override toString of KeyValue/Relati...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/19363
Please replace `[Minor]` with `[SQL]`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [Minor]Override toString of KeyValueGroupedDataset
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/19363
Btw, I think this might need a minor/trivial JIRA ticket.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][Minor]Override toString of KeyValue/Relati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19363
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19363: [SPARK-22224][SQL]Override toString of KeyValue/R...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19363#discussion_r144753514
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala ---
@@ -46,6 +36,20 @@ abstract class QueryTest extends PlanTest {
Locale.setDefault(Locale.US)
/**
+ * Makes sure the answer matches the expected result
+ */
+ def checkString(expected: String, actual: String): Unit = {
+ if (expected != actual) {
+ fail(
+ "The actual string gives wrong result:\n\n" + sideBySide(
--- End diff --
it's kind of an overkill, the `toString` we are testing always return a single line, just `assert` is enough.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][SQL]Override toString of KeyValue/Relation...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19363
LGTM, pending jenkins
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][Minor]Override toString of KeyValue/Relati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19363
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19363: [SPARK-22224][SQL]Override toString of KeyValue/R...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19363#discussion_r144753090
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/RelationalGroupedDataset.scala ---
@@ -21,6 +21,7 @@ import java.util.Locale
import scala.collection.JavaConverters._
import scala.language.implicitConversions
+import scala.util.control.NonFatal
--- End diff --
ditto
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][SQL]Override toString of KeyValue/Relation...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19363
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19363: [Minor]Override toString of KeyValueGroupedDatase...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19363#discussion_r141424043
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -54,6 +55,14 @@ class KeyValueGroupedDataset[K, V] private[sql](
private def sparkSession = queryExecution.sparkSession
/**
+ * Returns the schema of this Dataset.
+ *
+ * @group basic
+ * @since 2.3.0
+ */
+ def schema: StructType = queryExecution.analyzed.schema
--- End diff --
Can you remove this?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][SQL]Override toString of KeyValue/Relation...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19363
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82828/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][SQL]Override toString of KeyValue/Relation...
Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/19363
hi @cloud-fan jenkins ok
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][Minor]Override toString of KeyValue/Relati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19363
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82548/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [Minor]Override toString of KeyValueGroupedDataset
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/19363
Although this `toString` in scala-shell looks good, when you print out directly, it might look weird because you just see:
```scala
[key: [value: string], value: [value: string]]
```
It's better to add prefix of the class name, like.
```scala
KeyValueGroupedDataset: [key: [value: string], value: [value: string]]
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19363: [SPARK-22224][SQL]Override toString of KeyValue/R...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19363#discussion_r144753122
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/RelationalGroupedDataset.scala ---
@@ -465,6 +466,19 @@ class RelationalGroupedDataset protected[sql](
Dataset.ofRows(df.sparkSession, plan)
}
+
+ override def toString: String = {
+ val builder = new StringBuilder
+ builder.append("RelationalGroupedDataset: [groupingBy: [")
--- End diff --
`groupingBy` -> `group by`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19363: [SPARK-22224][SQL]Override toString of KeyValue/R...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19363#discussion_r144753076
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -18,13 +18,13 @@
package org.apache.spark.sql
import scala.collection.JavaConverters._
+import scala.util.control.NonFatal
--- End diff --
unnecessary import
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][Minor]Override toString of KeyValue/Relati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19363
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19363: [SPARK-22224][SQL]Override toString of KeyValue/R...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19363#discussion_r144873610
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala ---
@@ -1341,8 +1341,69 @@ class DatasetSuite extends QueryTest with SharedSQLContext {
Seq(1).toDS().map(_ => ("", TestForTypeAlias.seqOfTupleTypeAlias)),
("", Seq((1, 1), (2, 2))))
}
+
+ test("Check RelationalGroupedDataset toString: Single data") {
+ val kvDataset = (1 to 3).toDF("id").groupBy("id")
+ val expected = "RelationalGroupedDataset: [" +
+ "group by: [id: int], value: [id: int], type: GroupByType]"
--- End diff --
a better idea:
* `group by` -> `grouping expressions`
* `type: GroupByType` -> `type: GroupBy`, i.e. remove the `Type` suffix.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][SQL]Override toString of KeyValue/Relation...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19363
**[Test build #82776 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82776/testReport)** for PR 19363 at commit [`9166640`](https://github.com/apache/spark/commit/91666403e287015d953c4b105df6bc35dae966c3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19363: [SPARK-22224][SQL]Override toString of KeyValue/Relation...
Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/19363
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19363: [SPARK-22224][SQL]Override toString of KeyValue/R...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19363#discussion_r144502598
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/RelationalGroupedDatasetSuite.scala ---
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.spark.sql.test.SharedSQLContext
+
+class RelationalGroupedDatasetSuite extends QueryTest with SharedSQLContext {
+ import testImplicits._
+
+ test("Check RelationalGroupedDataset toString: Single data") {
+ val kvDataset = (1 to 3).toDF("id").as[SingleData].groupBy("id")
--- End diff --
Is `SingleData` necessary?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org