You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by KhajaAsmath Mohammed <md...@gmail.com> on 2017/11/17 17:06:21 UTC

Struct Type

Hi,

I have following schema in dataframe and I want to extract key which
matches as MaxSpeed from the array and it's corresponding value of the key.

|-- tags: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- key: string (nullable = true)
 |    |    |-- value: string (nullable = true)

is there any way to achieve it in dataframe?

Thanks,
Asmath

Re: Struct Type

Posted by Jacek Laskowski <ja...@japila.pl>.

Hi,

Use explode function, filter operator and collect_list function.

Or "heavier" flatMap.

Pozdrawiam,
Jacek Laskowski
----
https://about.me/JacekLaskowski
Spark Structured Streaming https://bit.ly/spark-structured-streaming
Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski

On Fri, Nov 17, 2017 at 6:06 PM, KhajaAsmath Mohammed <
mdkhajaasmath@gmail.com> wrote:

> Hi,
>
> I have following schema in dataframe and I want to extract key which
> matches as MaxSpeed from the array and it's corresponding value of the key.
>
> |-- tags: array (nullable = true)
>  |    |-- element: struct (containsNull = true)
>  |    |    |-- key: string (nullable = true)
>  |    |    |-- value: string (nullable = true)
>
> is there any way to achieve it in dataframe?
>
> Thanks,
> Asmath
>