[jira] [Commented] (SPARK-27820) case insensitive resolver should be used in GetMapValue

["Hyukjin Kwon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-27820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16847229#comment-16847229 ] 

Hyukjin Kwon commented on SPARK-27820:
--------------------------------------

I don't think we should make the case sensitivity works with maps. I think this option is supposed to be used for table identifier and column names but not for unique keys in maps. At least making case insensitive will break other users app.

> case insensitive resolver should be used in GetMapValue
> -------------------------------------------------------
>
>                 Key: SPARK-27820
>                 URL: https://issues.apache.org/jira/browse/SPARK-27820
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.4.1
>            Reporter: Michel Lemay
>            Priority: Minor
>
> When extracting a key value from a MapType, it calls GetMapValue (complexTypeExtractors.scala) and only use the map type ordering. It should use the resolver instead.
> Starting spark with: `{{spark-shell --conf spark.sql.caseSensitive=false`}}
> Given dataframe:
>  {{val df = List(Map("a" -> 1), Map("A" -> 2)).toDF("m")}}
> And executing any of these will only return one row: case insensitive in the name of the column but case sensitive match in the keys of the map.
> {{df.filter($"M.A".isNotNull).count}}
>  {{df.filter($"M"("A").isNotNull).count df.filter($"M".getField("A").isNotNull).count}}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org