You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by kupferk <gi...@git.apache.org> on 2018/07/06 05:15:08 UTC
[GitHub] spark pull request #21722: Spark-24742: Fix NullPointerexception in Field Me...
GitHub user kupferk opened a pull request:
https://github.com/apache/spark/pull/21722
Spark-24742: Fix NullPointerexception in Field Metadata
## What changes were proposed in this pull request?
This pull request provides a fix for SPARK-24742: SQL Field MetaData was throwing an Exception in the hashCode method when a "null" Metadata was added via "putNull"
## How was this patch tested?
A new unittest is provided in org/apache/spark/sql/types/MetadataSuite.scala
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kupferk/spark SPARK-24742
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21722.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21722
----
commit 08ad908728f6def2c07368a5cdd89df00f32ba2d
Author: Kaya Kupferschmidt <k....@...>
Date: 2018-07-05T06:01:28Z
SPARK-24742: Fixed NullPointerException in Metadata when storing null values
commit 088e2d789dad707bd657a72afa8933e957641536
Author: Kaya Kupferschmidt <k....@...>
Date: 2018-07-05T06:03:33Z
SPARK-24742: Improved TestSuite for Metadata
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21722: Spark-24742: Fix NullPointerexception in Field Me...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21722#discussion_r201180956
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Metadata.scala ---
@@ -215,6 +215,8 @@ object Metadata {
x.##
case x: Metadata =>
hash(x.map)
+ case null =>
--- End diff --
Basically, we do not support `null` in Metadata.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21722: Spark-24742: Fix NullPointerexception in Field Metadata
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21722
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21722: Spark-24742: Fix NullPointerexception in Field Me...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/21722#discussion_r206947926
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/types/MetadataSuite.scala ---
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.types
+
+import org.apache.spark.SparkFunSuite
+
+class MetadataSuite extends SparkFunSuite {
+ test("String Metadata") {
--- End diff --
Oh @kupferk actually just noticed that the indents in this file are too deep. They should be 2-space. I think it might also be useful to use `===` and `!==` as test failures produces more detailed error messages then.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21722: Spark-24742: Fix NullPointerexception in Field Me...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21722#discussion_r201181931
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Metadata.scala ---
@@ -215,6 +215,8 @@ object Metadata {
x.##
case x: Metadata =>
hash(x.map)
+ case null =>
--- End diff --
nvm, I found we added the support in https://github.com/apache/spark/pull/8969
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21722: Spark-24742: Fix NullPointerexception in Field Metadata
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21722
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21722: Spark-24742: Fix NullPointerexception in Field Metadata
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/21722
Heh, collided yeah. I thought one commit would end up failing but since they're identical guess it results in an empty commit. No big deal.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21722: Spark-24742: Fix NullPointerexception in Field Metadata
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21722
ooops.. I merged this master too and looks there's an empty commit by that ... - https://github.com/apache/spark/commit/d182b3d34d6afade401b8a455b774059bae9d90f
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21722: Spark-24742: Fix NullPointerexception in Field Metadata
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21722
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21722: Spark-24742: Fix NullPointerexception in Field Metadata
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21722
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21722: Spark-24742: Fix NullPointerexception in Field Me...
Posted by kupferk <gi...@git.apache.org>.
Github user kupferk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21722#discussion_r201219153
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Metadata.scala ---
@@ -215,6 +215,8 @@ object Metadata {
x.##
case x: Metadata =>
hash(x.map)
+ case null =>
--- End diff --
Yes, the `putNull` method exists, but is actually not useable because of this bug. Having `null` in Metadata would be really helpful for me, as I'd like to store default values for fields in the Metadata. And NULL is a very common case as a default value.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21722: Spark-24742: Fix NullPointerexception in Field Metadata
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21722
**[Test build #4228 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4228/testReport)** for PR 21722 at commit [`088e2d7`](https://github.com/apache/spark/commit/088e2d789dad707bd657a72afa8933e957641536).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21722: Spark-24742: Fix NullPointerexception in Field Metadata
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21722
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21722: Spark-24742: Fix NullPointerexception in Field Metadata
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21722
**[Test build #4227 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4227/testReport)** for PR 21722 at commit [`088e2d7`](https://github.com/apache/spark/commit/088e2d789dad707bd657a72afa8933e957641536).
* This patch **fails Spark unit tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21722: Spark-24742: Fix NullPointerexception in Field Metadata
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21722
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93978/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21722: Spark-24742: Fix NullPointerexception in Field Metadata
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21722
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21722: Spark-24742: Fix NullPointerexception in Field Metadata
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/21722
Merged to master
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21722: Spark-24742: Fix NullPointerexception in Field Metadata
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21722
Merged to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21722: Spark-24742: Fix NullPointerexception in Field Metadata
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21722
**[Test build #4227 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4227/testReport)** for PR 21722 at commit [`088e2d7`](https://github.com/apache/spark/commit/088e2d789dad707bd657a72afa8933e957641536).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21722: Spark-24742: Fix NullPointerexception in Field Metadata
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21722
**[Test build #93933 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93933/testReport)** for PR 21722 at commit [`17485a7`](https://github.com/apache/spark/commit/17485a703f686441ee1569b3ccfd3547af13f397).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21722: Spark-24742: Fix NullPointerexception in Field Metadata
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21722
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93933/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21722: Spark-24742: Fix NullPointerexception in Field Metadata
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21722
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21722: Spark-24742: Fix NullPointerexception in Field Metadata
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21722
**[Test build #93978 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93978/testReport)** for PR 21722 at commit [`17485a7`](https://github.com/apache/spark/commit/17485a703f686441ee1569b3ccfd3547af13f397).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21722: Spark-24742: Fix NullPointerexception in Field Metadata
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21722
**[Test build #93933 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93933/testReport)** for PR 21722 at commit [`17485a7`](https://github.com/apache/spark/commit/17485a703f686441ee1569b3ccfd3547af13f397).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21722: Spark-24742: Fix NullPointerexception in Field Me...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/21722
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21722: Spark-24742: Fix NullPointerexception in Field Metadata
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21722
**[Test build #4228 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4228/testReport)** for PR 21722 at commit [`088e2d7`](https://github.com/apache/spark/commit/088e2d789dad707bd657a72afa8933e957641536).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21722: Spark-24742: Fix NullPointerexception in Field Me...
Posted by kupferk <gi...@git.apache.org>.
Github user kupferk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21722#discussion_r206950519
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/types/MetadataSuite.scala ---
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.types
+
+import org.apache.spark.SparkFunSuite
+
+class MetadataSuite extends SparkFunSuite {
+ test("String Metadata") {
--- End diff --
Thanks for your comment, I am happy to fix that :)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21722: Spark-24742: Fix NullPointerexception in Field Metadata
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21722
**[Test build #93978 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93978/testReport)** for PR 21722 at commit [`17485a7`](https://github.com/apache/spark/commit/17485a703f686441ee1569b3ccfd3547af13f397).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org