You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Sebastian Schere <ss...@gmail.com> on 2021/05/08 22:01:39 UTC

How to have map_from_arrays() in Spark 2.3

I have the following Dataframe in Spark 2.3, coming from a JSON file:

root
 |-- ext_attr: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- key: string (nullable = true)
 |    |    |-- value: string (nullable = true)

I would need to convert it to below DataFrame:

root
 |-- ext_attr_map: map (nullable = false)
 |    |-- key: string
 |    |-- value: string

I know that in Spark 2.4 I'd use map_from_arrays() SQL function.

How can I achieve this in Spark 2.3 ? Please provide a UDF or SQL code.


Thanks,

Sebastian