[jira] [Comment Edited] (SPARK-20912) map function with columns as strings

["Takeshi Yamamuro (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-20912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16028180#comment-16028180 ] 

Takeshi Yamamuro edited comment on SPARK-20912 at 5/29/17 9:56 AM:
-------------------------------------------------------------------

You meant this?
{code}
scala> kvs.withColumn("map", map(lit("k"), lit("v"))).show
+---+-----+-----------+
|  k|    v|        map|
+---+-----+-----------+
|key|value|Map(k -> v)|
+---+-----+-----------+
{code}


was (Author: maropu):
You mean this?
{code}
scala> kvs.withColumn("map", map(lit("k"), lit("v"))).show
+---+-----+-----------+
|  k|    v|        map|
+---+-----+-----------+
|key|value|Map(k -> v)|
+---+-----+-----------+
{code}

> map function with columns as strings
> ------------------------------------
>
>                 Key: SPARK-20912
>                 URL: https://issues.apache.org/jira/browse/SPARK-20912
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 2.3.0
>            Reporter: Jacek Laskowski
>            Priority: Trivial
>
> There's only {{map}} function that accepts {{Column}} values only. It'd be very helpful to have a variant that accepted {{String}} for columns like {{array}} or {{struct}}.
> {code}
> scala> val kvs = Seq(("key", "value")).toDF("k", "v")
> kvs: org.apache.spark.sql.DataFrame = [k: string, v: string]
> scala> kvs.printSchema
> root
>  |-- k: string (nullable = true)
>  |-- v: string (nullable = true)
> scala> kvs.withColumn("map", map("k", "v")).show
> <console>:26: error: type mismatch;
>  found   : String("k")
>  required: org.apache.spark.sql.Column
>        kvs.withColumn("map", map("k", "v")).show
>                                  ^
> <console>:26: error: type mismatch;
>  found   : String("v")
>  required: org.apache.spark.sql.Column
>        kvs.withColumn("map", map("k", "v")).show
>                                       ^
> // note $ to create Columns per string
> // not very dev-friendly
> scala> kvs.withColumn("map", map($"k", $"v")).show
> +---+-----+-----------------+
> |  k|    v|              map|
> +---+-----+-----------------+
> |key|value|Map(key -> value)|
> +---+-----+-----------------+
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org