You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Divya Gehlot <di...@gmail.com> on 2016/07/18 07:23:34 UTC
Dynamically get value based on Map key in Spark Dataframe
Hi,
I have created a map by reading a text file
val keyValueMap = file_read.map(t => t.getString(0) ->
t.getString(4)).collect().toMap
Now I have another dataframe where I need to dynamically replace all the
keys of Map with values
val df_input = reading the file as dataframe
val df_replacekeys =
df_input.withColumn("map_values",lit(keyValueMap (col("key"))))
Would really appreciate the help .
Thanks,
Divya
Re: Dynamically get value based on Map key in Spark Dataframe
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi Divya,
That's the right way to access a value for a key in a broadcast map.
I'm pretty sure tough that you could do the same (or similar) with
higher-level broadcast Datasets. Try it out!
Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
On Tue, Jul 19, 2016 at 4:31 AM, Divya Gehlot <di...@gmail.com> wrote:
> Hi Jacek,
>
> Can you please share example how can I access broacasted map
> val pltStnMapBrdcst = sc.broadcast(keyvalueMap )
> val df_replacekeys =
> df_input.withColumn("map_values",pltStnMapBrdcst.value.get("key"))))
>
> Is the above the right way to access the broadcasted map ?
>
>
>
> Thanks,
> Divya
>
>
> On 18 July 2016 at 23:06, Jacek Laskowski <ja...@japila.pl> wrote:
>>
>> See broadcast variable.
>>
>> Or (just a thought) do join between DataFrames.
>>
>> Jacek
>>
>>
>> On 18 Jul 2016 9:24 a.m., "Divya Gehlot" <di...@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> I have created a map by reading a text file
>>> val keyValueMap = file_read.map(t => t.getString(0) ->
>>> t.getString(4)).collect().toMap
>>>
>>> Now I have another dataframe where I need to dynamically replace all the
>>> keys of Map with values
>>> val df_input = reading the file as dataframe
>>> val df_replacekeys = df_input.withColumn("map_values",lit(keyValueMap
>>> (col("key"))))
>>>
>>> Would really appreciate the help .
>>>
>>>
>>> Thanks,
>>> Divya
>>>
>>>
>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Dynamically get value based on Map key in Spark Dataframe
Posted by Divya Gehlot <di...@gmail.com>.
Hi Jacek,
Can you please share example how can I access broacasted map
val pltStnMapBrdcst = sc.broadcast(keyvalueMap )
val df_replacekeys = df_input.withColumn("map_values",
pltStnMapBrdcst.value.get("key"))))
Is the above the right way to access the broadcasted map ?
Thanks,
Divya
On 18 July 2016 at 23:06, Jacek Laskowski <ja...@japila.pl> wrote:
> See broadcast variable.
>
> Or (just a thought) do join between DataFrames.
>
> Jacek
>
> On 18 Jul 2016 9:24 a.m., "Divya Gehlot" <di...@gmail.com> wrote:
>
>> Hi,
>>
>> I have created a map by reading a text file
>> val keyValueMap = file_read.map(t => t.getString(0) ->
>> t.getString(4)).collect().toMap
>>
>> Now I have another dataframe where I need to dynamically replace all the
>> keys of Map with values
>> val df_input = reading the file as dataframe
>> val df_replacekeys =
>> df_input.withColumn("map_values",lit(keyValueMap (col("key"))))
>>
>> Would really appreciate the help .
>>
>>
>> Thanks,
>> Divya
>>
>>
>>
Re: Dynamically get value based on Map key in Spark Dataframe
Posted by Jacek Laskowski <ja...@japila.pl>.
See broadcast variable.
Or (just a thought) do join between DataFrames.
Jacek
On 18 Jul 2016 9:24 a.m., "Divya Gehlot" <di...@gmail.com> wrote:
> Hi,
>
> I have created a map by reading a text file
> val keyValueMap = file_read.map(t => t.getString(0) ->
> t.getString(4)).collect().toMap
>
> Now I have another dataframe where I need to dynamically replace all the
> keys of Map with values
> val df_input = reading the file as dataframe
> val df_replacekeys =
> df_input.withColumn("map_values",lit(keyValueMap (col("key"))))
>
> Would really appreciate the help .
>
>
> Thanks,
> Divya
>
>
>