You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by viirya <gi...@git.apache.org> on 2018/10/21 13:00:31 UTC
[GitHub] spark pull request #22787: [SPARK-25040][SQL] Empty string for non string ty...
GitHub user viirya opened a pull request:
https://github.com/apache/spark/pull/22787
[SPARK-25040][SQL] Empty string for non string types should be disallowed
## What changes were proposed in this pull request?
This takes over original PR at #22019. The original proposal is to have null for float and double types. Later a more reasonable proposal is to disallow empty strings. This patch adds logic to throw exception when finding empty strings for non string types.
## How was this patch tested?
Added test.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/viirya/spark-1 SPARK-25040
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22787.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22787
----
commit 45aacaf916826eb8509c85c056b985172d0e2f50
Author: Liang-Chi Hsieh <vi...@...>
Date: 2018-10-21T12:57:26Z
Disallow empty strings on non string types.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22787
**[Test build #97695 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97695/testReport)** for PR 22787 at commit [`589caf7`](https://github.com/apache/spark/commit/589caf790c926eabb782d4a580ad524b5f6be74f).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22787
**[Test build #97688 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97688/testReport)** for PR 22787 at commit [`bb117f2`](https://github.com/apache/spark/commit/bb117f25dd1bd2e47a1536779d0641a44fb0da8f).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22787
github looks buggy for now. Let me clean up my comments if they got messed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22787
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97685/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22787
Yea, looks we should better fix the comments.
LGTM otherwise.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22787
**[Test build #97757 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97757/testReport)** for PR 22787 at commit [`bb117f2`](https://github.com/apache/spark/commit/bb117f25dd1bd2e47a1536779d0641a44fb0da8f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22787
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97692/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22787
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22787
**[Test build #97762 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97762/testReport)** for PR 22787 at commit [`589caf7`](https://github.com/apache/spark/commit/589caf790c926eabb782d4a580ad524b5f6be74f).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22787
**[Test build #97869 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97869/testReport)** for PR 22787 at commit [`c04ea64`](https://github.com/apache/spark/commit/c04ea649a9087f9cb5ffbb7361fbb711d74d9266).
* This patch **fails to generate documentation**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22787
It's chaotic ...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22787
Seems github is restored...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22787
**[Test build #97775 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97775/testReport)** for PR 22787 at commit [`589caf7`](https://github.com/apache/spark/commit/589caf790c926eabb782d4a580ad524b5f6be74f).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22787
**[Test build #97695 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97695/testReport)** for PR 22787 at commit [`589caf7`](https://github.com/apache/spark/commit/589caf790c926eabb782d4a580ad524b5f6be74f).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22787
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4259/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22787
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22787
**[Test build #97684 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97684/testReport)** for PR 22787 at commit [`45aacaf`](https://github.com/apache/spark/commit/45aacaf916826eb8509c85c056b985172d0e2f50).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22787
**[Test build #97690 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97690/testReport)** for PR 22787 at commit [`589caf7`](https://github.com/apache/spark/commit/589caf790c926eabb782d4a580ad524b5f6be74f).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22787
retest this please.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22787
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22787
**[Test build #97832 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97832/testReport)** for PR 22787 at commit [`c04ea64`](https://github.com/apache/spark/commit/c04ea649a9087f9cb5ffbb7361fbb711d74d9266).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22787
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22787
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22787
**[Test build #97858 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97858/testReport)** for PR 22787 at commit [`c04ea64`](https://github.com/apache/spark/commit/c04ea649a9087f9cb5ffbb7361fbb711d74d9266).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22787
**[Test build #97684 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97684/testReport)** for PR 22787 at commit [`45aacaf`](https://github.com/apache/spark/commit/45aacaf916826eb8509c85c056b985172d0e2f50).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22787
**[Test build #97757 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97757/testReport)** for PR 22787 at commit [`bb117f2`](https://github.com/apache/spark/commit/bb117f25dd1bd2e47a1536779d0641a44fb0da8f).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22787: [SPARK-25040][SQL] Empty string for non string ty...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/22787
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22787
**[Test build #97843 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97843/testReport)** for PR 22787 at commit [`c04ea64`](https://github.com/apache/spark/commit/c04ea649a9087f9cb5ffbb7361fbb711d74d9266).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22787
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22787
**[Test build #97769 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97769/testReport)** for PR 22787 at commit [`bb117f2`](https://github.com/apache/spark/commit/bb117f25dd1bd2e47a1536779d0641a44fb0da8f).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22787
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97884/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22787
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4378/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22787
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22787
**[Test build #97692 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97692/testReport)** for PR 22787 at commit [`bb117f2`](https://github.com/apache/spark/commit/bb117f25dd1bd2e47a1536779d0641a44fb0da8f).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22787
**[Test build #97685 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97685/testReport)** for PR 22787 at commit [`589caf7`](https://github.com/apache/spark/commit/589caf790c926eabb782d4a580ad524b5f6be74f).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22787
**[Test build #97884 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97884/testReport)** for PR 22787 at commit [`c04ea64`](https://github.com/apache/spark/commit/c04ea649a9087f9cb5ffbb7361fbb711d74d9266).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22787
**[Test build #97884 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97884/testReport)** for PR 22787 at commit [`c04ea64`](https://github.com/apache/spark/commit/c04ea649a9087f9cb5ffbb7361fbb711d74d9266).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22787
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4181/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22787
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4213/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22787
**[Test build #97864 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97864/testReport)** for PR 22787 at commit [`bb117f2`](https://github.com/apache/spark/commit/bb117f25dd1bd2e47a1536779d0641a44fb0da8f).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22787
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97688/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22787
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4257/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22787
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22787
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22787
cc @cloud-fan
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22787: [SPARK-25040][SQL] Empty string for non string ty...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/22787#discussion_r226884659
--- Diff: docs/sql-migration-guide-upgrade.md ---
@@ -11,6 +11,8 @@ displayTitle: Spark SQL Upgrading Guide
- In PySpark, when creating a `SparkSession` with `SparkSession.builder.getOrCreate()`, if there is an existing `SparkContext`, the builder was trying to update the `SparkConf` of the existing `SparkContext` with configurations specified to the builder, but the `SparkContext` is shared by all `SparkSession`s, so we should not update them. Since 3.0, the builder comes to not update the configurations. This is the same behavior as Java/Scala API in 2.3 and above. If you want to update them, you need to update them prior to creating a `SparkSession`.
+ - In Spark version 2.4 and earlier, the parser of JSON data source treats empty strings as null for some data types like `IntegerType`. For `FloatType` and `DoubleType`, it fails on empty strings and throws exceptions. Since Spark 3.0, we disallow empty strings and will throw exceptions for data types except for `StringType` and `BinaryType`.
--- End diff --
Thanks. I've updated it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22787
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4217/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22787
cc @HyukjinKwon
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22787
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22787
Yea, I do think so. I will update it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22787
Thanks @HyukjinKwon @dongjoon-hyun
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22787
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97695/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22787
**[Test build #97837 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97837/testReport)** for PR 22787 at commit [`bb117f2`](https://github.com/apache/spark/commit/bb117f25dd1bd2e47a1536779d0641a44fb0da8f).
* This patch **fails to generate documentation**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22787: [SPARK-25040][SQL] Empty string for non string ty...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22787#discussion_r226871288
--- Diff: docs/sql-migration-guide-upgrade.md ---
@@ -11,6 +11,8 @@ displayTitle: Spark SQL Upgrading Guide
- In PySpark, when creating a `SparkSession` with `SparkSession.builder.getOrCreate()`, if there is an existing `SparkContext`, the builder was trying to update the `SparkConf` of the existing `SparkContext` with configurations specified to the builder, but the `SparkContext` is shared by all `SparkSession`s, so we should not update them. Since 3.0, the builder comes to not update the configurations. This is the same behavior as Java/Scala API in 2.3 and above. If you want to update them, you need to update them prior to creating a `SparkSession`.
+ - In Spark version 2.4 and earlier, the parser of JSON data source treats empty strings as null for some data types like `IntegerType`. For `FloatType` and `DoubleType`, it fails on empty strings and throws exceptions. Since Spark 3.0, we disallow empty strings and will throw exceptions for data types except for `StringType` and `BinaryType`.
--- End diff --
In the rest of this PR, we are saying `non string type` while we are mentioning `StringType` and `BinaryType` in this migration doc. Do we need to mention `binary` type in the following lines, too?
- https://github.com/apache/spark/pull/22787/files#diff-635e02b2d1ce4ad1675b0350ccac0c10R313
- https://github.com/apache/spark/pull/22787/files#diff-635e02b2d1ce4ad1675b0350ccac0c10R320
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22787: [SPARK-25040][SQL] Empty string for non string ty...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/22787#discussion_r226884649
--- Diff: docs/sql-migration-guide-upgrade.md ---
@@ -11,6 +11,8 @@ displayTitle: Spark SQL Upgrading Guide
- In PySpark, when creating a `SparkSession` with `SparkSession.builder.getOrCreate()`, if there is an existing `SparkContext`, the builder was trying to update the `SparkConf` of the existing `SparkContext` with configurations specified to the builder, but the `SparkContext` is shared by all `SparkSession`s, so we should not update them. Since 3.0, the builder comes to not update the configurations. This is the same behavior as Java/Scala API in 2.3 and above. If you want to update them, you need to update them prior to creating a `SparkSession`.
+ - In Spark version 2.4 and earlier, the parser of JSON data source treats empty strings as null for some data types like `IntegerType`. For `FloatType` and `DoubleType`, it fails on empty strings and throws exceptions. Since Spark 3.0, we disallow empty strings and will throw exceptions for data types except for `StringType` and `BinaryType`.
--- End diff --
Thanks. I've updated it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22787
Yea looks good as we discussed. Should we maybe better update the migration guide too while we are here?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22787
**[Test build #97762 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97762/testReport)** for PR 22787 at commit [`589caf7`](https://github.com/apache/spark/commit/589caf790c926eabb782d4a580ad524b5f6be74f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22787
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22787
Thanks @HyukjinKwon.
I've fixed that and committed. But looks like Github is out of order now.
The commit is now disappeared. Let's wait for Github backing to normal. :)
On Mon, Oct 22, 2018 at 12:50 PM Hyukjin Kwon <no...@github.com>
wrote:
> *@HyukjinKwon* approved this pull request.
>
> Yea, looks we should fix the comments.
>
> LGTM otherwise
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/spark/pull/22787#pullrequestreview-166823987>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAEM9_r0kwKh8oV3uNoOLB59LU_Kwtgnks5unU6KgaJpZM4Xycnv>
> .
>
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22787
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97690/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22787
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97684/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22787
**[Test build #97685 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97685/testReport)** for PR 22787 at commit [`589caf7`](https://github.com/apache/spark/commit/589caf790c926eabb782d4a580ad524b5f6be74f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22787
**[Test build #97688 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97688/testReport)** for PR 22787 at commit [`bb117f2`](https://github.com/apache/spark/commit/bb117f25dd1bd2e47a1536779d0641a44fb0da8f).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22787
**[Test build #97690 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97690/testReport)** for PR 22787 at commit [`589caf7`](https://github.com/apache/spark/commit/589caf790c926eabb782d4a580ad524b5f6be74f).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22787
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22787
Merged to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22787
**[Test build #97852 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97852/testReport)** for PR 22787 at commit [`bb117f2`](https://github.com/apache/spark/commit/bb117f25dd1bd2e47a1536779d0641a44fb0da8f).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22787
**[Test build #97692 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97692/testReport)** for PR 22787 at commit [`bb117f2`](https://github.com/apache/spark/commit/bb117f25dd1bd2e47a1536779d0641a44fb0da8f).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22787
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22787
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22787: [SPARK-25040][SQL] Empty string for non string types sho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22787
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4180/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org