You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by xysun <gi...@git.apache.org> on 2018/02/21 09:34:22 UTC
[GitHub] spark pull request #20649: [SPARK-23452][SQL] improve missing field error me...
GitHub user xysun opened a pull request:
https://github.com/apache/spark/pull/20649
[SPARK-23452][SQL] improve missing field error message in `StructType`
## What changes were proposed in this pull request?
The error message ```s"""Field "$name" does not exist."""``` is thrown when looking up an unknown field in StructType. In the error message, we should also contain the information about which columns/fields exist in this struct.
## How was this patch tested?
Added new unit tests.
Note: I created a new `StructTypeSuite.scala` as I couldn't find an existing suite that's suitable to place these tests. I may be missing something so feel free to propose new locations.
Please review http://spark.apache.org/contributing.html before opening a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xysun/spark SPARK-23462
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20649.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20649
----
commit e1a5ccfdfc4ea4355207cd121a372ad605a0fc3d
Author: Xiayun Sun <xi...@...>
Date: 2018-02-21T09:31:11Z
output current fields in missing fields error message
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by xysun <gi...@git.apache.org>.
Github user xysun commented on the issue:
https://github.com/apache/spark/pull/20649
:( can anyone help issue the retest command? @mgaido91 @gatorsmile thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20649: [SPARK-23462][SQL] improve missing field error me...
Posted by xysun <gi...@git.apache.org>.
Github user xysun commented on a diff in the pull request:
https://github.com/apache/spark/pull/20649#discussion_r172104069
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala ---
@@ -284,7 +286,8 @@ case class StructType(fields: Array[StructField]) extends DataType with Seq[Stru
val nonExistFields = names -- fieldNamesSet
if (nonExistFields.nonEmpty) {
throw new IllegalArgumentException(
- s"Field ${nonExistFields.mkString(",")} does not exist.")
+ s"""Fields ${nonExistFields.mkString(", ")} does not exist.
--- End diff --
done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20649
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87599/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20649
Merged to master and branch-2.3.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20649
**[Test build #87792 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87792/testReport)** for PR 20649 at commit [`8cdb1d5`](https://github.com/apache/spark/commit/8cdb1d52117325fcbdd1cefc9e9f0616afdb2baa).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class StructTypeSuite extends SparkFunSuite `
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20649
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87608/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20649
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87952/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20649: [SPARK-23462][SQL] improve missing field error me...
Posted by xysun <gi...@git.apache.org>.
Github user xysun commented on a diff in the pull request:
https://github.com/apache/spark/pull/20649#discussion_r171182024
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/StructTypeSuite.scala ---
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.types.StructType
+
+class StructTypeSuite extends SparkFunSuite {
+
+ test("SPARK-23462 lookup a single missing field should output existing fields") {
+ val s = StructType.fromDDL("a INT, b STRING")
+ val e = intercept[IllegalArgumentException](s("c")).getMessage
+ assert(e.contains("Available fields: a,b"))
+ }
+
+ test("SPARK-23462 lookup a set of missing fields should output existing fields") {
+ val s = StructType.fromDDL("a INT, b STRING")
+ val e = intercept[IllegalArgumentException](s(Set("a", "c"))).getMessage
+ assert(e.contains("Available fields: a,b"))
+ }
+
+ test("SPARK-23462 lookup fieldIndex for missing field should output existing fields") {
+ val s = StructType.fromDDL("a INT, b STRING")
+ val e = intercept[IllegalArgumentException](s.fieldIndex("c")).getMessage
+ assert(e.contains("Available fields: a,b"))
+ }
+
--- End diff --
all done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20649
**[Test build #87759 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87759/testReport)** for PR 20649 at commit [`8cdb1d5`](https://github.com/apache/spark/commit/8cdb1d52117325fcbdd1cefc9e9f0616afdb2baa).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class StructTypeSuite extends SparkFunSuite `
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20649: [SPARK-23462][SQL] improve missing field error me...
Posted by xysun <gi...@git.apache.org>.
Github user xysun commented on a diff in the pull request:
https://github.com/apache/spark/pull/20649#discussion_r172084508
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala ---
@@ -284,7 +286,8 @@ case class StructType(fields: Array[StructField]) extends DataType with Seq[Stru
val nonExistFields = names -- fieldNamesSet
if (nonExistFields.nonEmpty) {
throw new IllegalArgumentException(
- s"Field ${nonExistFields.mkString(",")} does not exist.")
+ s"""Fields ${nonExistFields.mkString(", ")} does not exist.
--- End diff --
"Field(s) ... do(es) not exist." like this?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20649: [SPARK-23462][SQL] improve missing field error me...
Posted by xysun <gi...@git.apache.org>.
Github user xysun commented on a diff in the pull request:
https://github.com/apache/spark/pull/20649#discussion_r169850853
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/StructTypeSuite.scala ---
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.types.StructType
+
+class StructTypeSuite extends SparkFunSuite{
+
+ test("SPARK-23462 lookup a single missing field should output existing fields") {
+ val s = StructType.fromDDL("a INT")
--- End diff --
done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20649: [SPARK-23462][SQL] improve missing field error me...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/20649
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20649: [SPARK-23462][SQL] improve missing field error me...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/20649#discussion_r173071330
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala ---
@@ -271,7 +271,9 @@ case class StructType(fields: Array[StructField]) extends DataType with Seq[Stru
*/
def apply(name: String): StructField = {
nameToField.getOrElse(name,
- throw new IllegalArgumentException(s"""Field "$name" does not exist."""))
+ throw new IllegalArgumentException(
+ s"""Field "$name" does not exist.
+ |Available fields: ${fieldNamesSet.mkString(", ")}""".stripMargin))
--- End diff --
shall we use `fieldNames` to keep the field order?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20649: [SPARK-23462][SQL] improve missing field error me...
Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20649#discussion_r170256168
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala ---
@@ -297,7 +300,9 @@ case class StructType(fields: Array[StructField]) extends DataType with Seq[Stru
*/
def fieldIndex(name: String): Int = {
nameToIndex.getOrElse(name,
- throw new IllegalArgumentException(s"""Field "$name" does not exist."""))
+ throw new IllegalArgumentException(
+ s"""Field "$name" does not exist.
+ |Available fields: ${fieldNamesSet.mkString(",")}""".stripMargin))
--- End diff --
ditto
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by xysun <gi...@git.apache.org>.
Github user xysun commented on the issue:
https://github.com/apache/spark/pull/20649
hey @HyukjinKwon @gatorsmile @mgaido91 anything else you'd like to comment/change? else are we good to merge?
Thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/20649
LGTM, cc @gatorsmile @cloud-fan
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20649
**[Test build #87953 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87953/testReport)** for PR 20649 at commit [`b2d1888`](https://github.com/apache/spark/commit/b2d18887d7263912c66ca7a032e1d8d4d6b6de70).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20649: [SPARK-23462][SQL] improve missing field error me...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20649#discussion_r172080837
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala ---
@@ -284,7 +286,8 @@ case class StructType(fields: Array[StructField]) extends DataType with Seq[Stru
val nonExistFields = names -- fieldNamesSet
if (nonExistFields.nonEmpty) {
throw new IllegalArgumentException(
- s"Field ${nonExistFields.mkString(",")} does not exist.")
+ s"""Fields ${nonExistFields.mkString(", ")} does not exist.
--- End diff --
nit: `nonExistFields` can contain only one field, so maybe `Fields` -> `Field(s)`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20649
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87792/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by xysun <gi...@git.apache.org>.
Github user xysun commented on the issue:
https://github.com/apache/spark/pull/20649
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20649
ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20649
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20649
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20649
**[Test build #88166 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88166/testReport)** for PR 20649 at commit [`78e037d`](https://github.com/apache/spark/commit/78e037d06d2972062cac14fe8b503e0802a4c32e).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20649
**[Test build #87608 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87608/testReport)** for PR 20649 at commit [`22f89f8`](https://github.com/apache/spark/commit/22f89f8537948e68af52acc54d0ea1d979b40753).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20649
ping @xysun
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20649
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20649: [SPARK-23462][SQL] improve missing field error me...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/20649#discussion_r169840705
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/StructTypeSuite.scala ---
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.types.StructType
+
+class StructTypeSuite extends SparkFunSuite{
+
+ test("SPARK-23462 lookup a single missing field should output existing fields") {
+ val s = StructType.fromDDL("a INT")
--- End diff --
add one more column?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23452][SQL] improve missing field error message i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20649
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by xysun <gi...@git.apache.org>.
Github user xysun commented on the issue:
https://github.com/apache/spark/pull/20649
I'm getting some weird compile errors on local even when based off previous successful commit.. checking..
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20649
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20649: [SPARK-23462][SQL] improve missing field error me...
Posted by xysun <gi...@git.apache.org>.
Github user xysun commented on a diff in the pull request:
https://github.com/apache/spark/pull/20649#discussion_r171181994
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/StructTypeSuite.scala ---
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.types.StructType
+
+class StructTypeSuite extends SparkFunSuite {
--- End diff --
moved
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20649: [SPARK-23462][SQL] improve missing field error me...
Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20649#discussion_r170256030
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala ---
@@ -284,7 +286,8 @@ case class StructType(fields: Array[StructField]) extends DataType with Seq[Stru
val nonExistFields = names -- fieldNamesSet
if (nonExistFields.nonEmpty) {
throw new IllegalArgumentException(
- s"Field ${nonExistFields.mkString(",")} does not exist.")
+ s"""Fields ${nonExistFields.mkString(",")} does not exist.
+ |Available fields: ${fieldNamesSet.mkString(",")}""".stripMargin)
--- End diff --
ditto
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20649
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20649
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20649
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20649
**[Test build #87952 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87952/testReport)** for PR 20649 at commit [`b2d1888`](https://github.com/apache/spark/commit/b2d18887d7263912c66ca7a032e1d8d4d6b6de70).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by xysun <gi...@git.apache.org>.
Github user xysun commented on the issue:
https://github.com/apache/spark/pull/20649
Latest code review fixes pushed. cc @HyukjinKwon
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20649: [SPARK-23462][SQL] improve missing field error me...
Posted by xysun <gi...@git.apache.org>.
Github user xysun commented on a diff in the pull request:
https://github.com/apache/spark/pull/20649#discussion_r171182128
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/StructTypeSuite.scala ---
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.types.StructType
+
+class StructTypeSuite extends SparkFunSuite {
+
+ test("SPARK-23462 lookup a single missing field should output existing fields") {
--- End diff --
removed JIRA number.
@kiszk i still prefer to keep separate tests as they are calling separate methods. Let me know if you feel strongly against it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20649
**[Test build #88076 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88076/testReport)** for PR 20649 at commit [`b2d1888`](https://github.com/apache/spark/commit/b2d18887d7263912c66ca7a032e1d8d4d6b6de70).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20649
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87759/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20649
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20649: [SPARK-23462][SQL] improve missing field error me...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/20649#discussion_r169882987
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/StructTypeSuite.scala ---
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.types.StructType
+
+class StructTypeSuite extends SparkFunSuite {
--- End diff --
Could we put this file under `sql/catalyst/src/test/scala/org/apache/spark/sql/types` or move the test cases into `DataTypeSuite`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20649
**[Test build #87953 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87953/testReport)** for PR 20649 at commit [`b2d1888`](https://github.com/apache/spark/commit/b2d18887d7263912c66ca7a032e1d8d4d6b6de70).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20649: [SPARK-23462][SQL] improve missing field error me...
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/20649#discussion_r170265519
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/StructTypeSuite.scala ---
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.types.StructType
+
+class StructTypeSuite extends SparkFunSuite {
+
+ test("SPARK-23462 lookup a single missing field should output existing fields") {
--- End diff --
nit: Would it be better to combine these three tests into one? It could reduce the total test time.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20649: [SPARK-23462][SQL] improve missing field error me...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20649#discussion_r170452170
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/StructTypeSuite.scala ---
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.types.StructType
+
+class StructTypeSuite extends SparkFunSuite {
+
+ test("SPARK-23462 lookup a single missing field should output existing fields") {
--- End diff --
This is not a bug fix, so please don't include the JIRA number in the test case description.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23452][SQL] improve missing field error message i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20649
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20649: [SPARK-23462][SQL] improve missing field error me...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/20649#discussion_r169840682
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/StructTypeSuite.scala ---
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.types.StructType
+
+class StructTypeSuite extends SparkFunSuite{
--- End diff --
Add a space before `{`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20649: [SPARK-23462][SQL] improve missing field error me...
Posted by xysun <gi...@git.apache.org>.
Github user xysun commented on a diff in the pull request:
https://github.com/apache/spark/pull/20649#discussion_r169850843
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/StructTypeSuite.scala ---
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.types.StructType
+
+class StructTypeSuite extends SparkFunSuite{
--- End diff --
done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20649: [SPARK-23462][SQL] improve missing field error me...
Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20649#discussion_r170256000
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala ---
@@ -271,7 +271,9 @@ case class StructType(fields: Array[StructField]) extends DataType with Seq[Stru
*/
def apply(name: String): StructField = {
nameToField.getOrElse(name,
- throw new IllegalArgumentException(s"""Field "$name" does not exist."""))
+ throw new IllegalArgumentException(
+ s"""Field "$name" does not exist.
+ |Available fields: ${fieldNamesSet.mkString(",")}""".stripMargin))
--- End diff --
nit: can we add a space after the comma? I think it can help readability
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20649
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88166/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20649
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87953/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20649
**[Test build #88166 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88166/testReport)** for PR 20649 at commit [`78e037d`](https://github.com/apache/spark/commit/78e037d06d2972062cac14fe8b503e0802a4c32e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20649
**[Test build #88076 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88076/testReport)** for PR 20649 at commit [`b2d1888`](https://github.com/apache/spark/commit/b2d18887d7263912c66ca7a032e1d8d4d6b6de70).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20649
**[Test build #87792 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87792/testReport)** for PR 20649 at commit [`8cdb1d5`](https://github.com/apache/spark/commit/8cdb1d52117325fcbdd1cefc9e9f0616afdb2baa).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20649
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23452][SQL] improve missing field error message i...
Posted by xysun <gi...@git.apache.org>.
Github user xysun commented on the issue:
https://github.com/apache/spark/pull/20649
@gatorsmile please review. Thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20649
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88076/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20649
**[Test build #87599 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87599/testReport)** for PR 20649 at commit [`e1a5ccf`](https://github.com/apache/spark/commit/e1a5ccfdfc4ea4355207cd121a372ad605a0fc3d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20649: [SPARK-23462][SQL] improve missing field error me...
Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20649#discussion_r170256385
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/StructTypeSuite.scala ---
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.types.StructType
+
+class StructTypeSuite extends SparkFunSuite {
+
+ test("SPARK-23462 lookup a single missing field should output existing fields") {
+ val s = StructType.fromDDL("a INT, b STRING")
+ val e = intercept[IllegalArgumentException](s("c")).getMessage
+ assert(e.contains("Available fields: a,b"))
+ }
+
+ test("SPARK-23462 lookup a set of missing fields should output existing fields") {
+ val s = StructType.fromDDL("a INT, b STRING")
+ val e = intercept[IllegalArgumentException](s(Set("a", "c"))).getMessage
+ assert(e.contains("Available fields: a,b"))
+ }
+
+ test("SPARK-23462 lookup fieldIndex for missing field should output existing fields") {
+ val s = StructType.fromDDL("a INT, b STRING")
+ val e = intercept[IllegalArgumentException](s.fieldIndex("c")).getMessage
+ assert(e.contains("Available fields: a,b"))
+ }
+
--- End diff --
nit: this empty line should be removed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20649
I usually leave open it for few more days in case other reviewers have some more review comments.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20649: [SPARK-23462][SQL] improve missing field error me...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20649#discussion_r172086269
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala ---
@@ -284,7 +286,8 @@ case class StructType(fields: Array[StructField]) extends DataType with Seq[Stru
val nonExistFields = names -- fieldNamesSet
if (nonExistFields.nonEmpty) {
throw new IllegalArgumentException(
- s"Field ${nonExistFields.mkString(",")} does not exist.")
+ s"""Fields ${nonExistFields.mkString(", ")} does not exist.
--- End diff --
`Not existing field(s): ${nonExistFields.mkString(", ")}` ?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20649
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20649
**[Test build #87759 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87759/testReport)** for PR 20649 at commit [`8cdb1d5`](https://github.com/apache/spark/commit/8cdb1d52117325fcbdd1cefc9e9f0616afdb2baa).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20649
**[Test build #87599 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87599/testReport)** for PR 20649 at commit [`e1a5ccf`](https://github.com/apache/spark/commit/e1a5ccfdfc4ea4355207cd121a372ad605a0fc3d).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/20649
LGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20649
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20649
**[Test build #87952 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87952/testReport)** for PR 20649 at commit [`b2d1888`](https://github.com/apache/spark/commit/b2d18887d7263912c66ca7a032e1d8d4d6b6de70).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20649: [SPARK-23462][SQL] improve missing field error me...
Posted by xysun <gi...@git.apache.org>.
Github user xysun commented on a diff in the pull request:
https://github.com/apache/spark/pull/20649#discussion_r173682838
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala ---
@@ -271,7 +271,9 @@ case class StructType(fields: Array[StructField]) extends DataType with Seq[Stru
*/
def apply(name: String): StructField = {
nameToField.getOrElse(name,
- throw new IllegalArgumentException(s"""Field "$name" does not exist."""))
+ throw new IllegalArgumentException(
+ s"""Field "$name" does not exist.
+ |Available fields: ${fieldNamesSet.mkString(", ")}""".stripMargin))
--- End diff --
done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20649: [SPARK-23462][SQL] improve missing field error me...
Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20649#discussion_r170256107
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala ---
@@ -284,7 +286,8 @@ case class StructType(fields: Array[StructField]) extends DataType with Seq[Stru
val nonExistFields = names -- fieldNamesSet
if (nonExistFields.nonEmpty) {
throw new IllegalArgumentException(
- s"Field ${nonExistFields.mkString(",")} does not exist.")
+ s"""Fields ${nonExistFields.mkString(",")} does not exist.
--- End diff --
ditto
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20649
**[Test build #87608 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87608/testReport)** for PR 20649 at commit [`22f89f8`](https://github.com/apache/spark/commit/22f89f8537948e68af52acc54d0ea1d979b40753).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class StructTypeSuite extends SparkFunSuite `
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20649: [SPARK-23462][SQL] improve missing field error message i...
Posted by xysun <gi...@git.apache.org>.
Github user xysun commented on the issue:
https://github.com/apache/spark/pull/20649
hey @gatorsmile @kiszk @jiangxb1987 @HyukjinKwon any more comments please?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org