You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by zuowang <gi...@git.apache.org> on 2016/02/28 16:17:07 UTC
[GitHub] spark pull request: [SPARK-13493][SQL] Enable case sensitiveness i...
GitHub user zuowang opened a pull request:
https://github.com/apache/spark/pull/11420
[SPARK-13493][SQL] Enable case sensitiveness in json schema inference.
## What changes were proposed in this pull request?
Pass "caseSensitiveAnalysis" to InferSchema. Change field name to lowercase when "caseSensitiveAnalysis" is false.
## How was this patch tested?
Add new test case in DataFrameSuite.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zuowang/spark case-sensitiveness-4-json
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11420.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11420
----
commit 7c76be43e8148124a124058634c74f8573daec8e
Author: Zuo Wang <zu...@intel.com>
Date: 2016-02-28T06:04:38Z
[SPARK-13493][SQL] Enable case sensitiveness in json schema inference.
Pass "caseSensitiveAnalysis" to InferSchema. Change field name to lowercase when "caseSensitiveAnalysis" is false.
Add new test case in DataFrameSuite.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13493][SQL] Enable case sensitiveness i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11420#issuecomment-189891409
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #11420: [SPARK-13493][SQL] Enable case sensitiveness in json sch...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/11420
I'm not, so it's okay to do so.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13493][SQL] Enable case sensitiveness i...
Posted by tedyu <gi...@git.apache.org>.
Github user tedyu commented on the pull request:
https://github.com/apache/spark/pull/11420#issuecomment-189905340
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #11420: [SPARK-13493][SQL] Enable case sensitiveness in json sch...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/11420
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #11420: [SPARK-13493][SQL] Enable case sensitiveness in json sch...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/11420
@rxin Currently, do we have any policy to handle case-sensitivity inside spark? e.g., `postgresql` always holds names into lower case (https://www.postgresql.org/docs/9.5/static/sql-syntax-lexical.html) and `h2` does into upper one. It seems there are many case-sensitivity issue in not only `json`, but also other other datasources.
unquoted names are always folded to lower case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #11420: [SPARK-13493][SQL] Enable case sensitiveness in json sch...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/11420
Let's close this if it is not in a progress.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #11420: [SPARK-13493][SQL] Enable case sensitiveness in j...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/11420
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #11420: [SPARK-13493][SQL] Enable case sensitiveness in json sch...
Posted by zuowang <gi...@git.apache.org>.
Github user zuowang commented on the issue:
https://github.com/apache/spark/pull/11420
I'm not, could you work to resolve it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #11420: [SPARK-13493][SQL] Enable case sensitiveness in json sch...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/11420
@zuowang @maropu Are you still working on this? We should resolve this issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #11420: [SPARK-13493][SQL] Enable case sensitiveness in json sch...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/11420
We are closing it due to inactivity. please do reopen if you want to push it forward. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13493][SQL] Enable case sensitiveness i...
Posted by zuowang <gi...@git.apache.org>.
Github user zuowang commented on the pull request:
https://github.com/apache/spark/pull/11420#issuecomment-190519404
What would be the case like in CSV? I don't think anybody want to conbine "user" and "User" column when "caseSensitiveAnalysis" is false.
`scala> val jsonDF = sqlContext.read.option("header", "true").csv("file.csv")
jsonDF: org.apache.spark.sql.DataFrame = [user: string, User: string ... 1 more field]
scala> jsonDF.show
+-------+-------+-----+
| user| User| hits|
+-------+-------+-----+
| om| scala| 120|
| daniel| spark| 80|
|3754978| spark| 1|
+-------+-------+-----+`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #11420: [SPARK-13493][SQL] Enable case sensitiveness in json sch...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/11420
@maropu Yes. We need to consider case sensitivity for the built-in data sources at least.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org