You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Soheil Pourbafrani <so...@gmail.com> on 2018/06/19 17:38:20 UTC
How can I do the following simple scenario in spark
Hi, I have a JSON file in the following structure:
+--------------------+-------------------+
| full_text| id|
+--------------------+-------------------+
I want to tokenize each sentence into pairs of (word, id)
for example, having the record : ("Hi, How are you?", id) I want to convert
the dataframe to:
hi, id
how, id
are, id
you, id
?, id
So I try :
data.rdd.map(lambda data : (data[0], data[1]))\
.flatMap(lambda row: (word_tokenize(row[0].lower()), row[1])
but it converts the dataframe to:
[hi, how, are, you, ?]
How can I do the desired transformation?
Re: How can I do the following simple scenario in spark
Posted by Sonal Goyal <so...@gmail.com>.
Try flatMapToPair instead of flatMap
Thanks,
Sonal
Nube Technologies <http://www.nubetech.co>
<http://in.linkedin.com/in/sonalgoyal>
On Tue, Jun 19, 2018 at 11:08 PM, Soheil Pourbafrani <so...@gmail.com>
wrote:
> Hi, I have a JSON file in the following structure:
> +--------------------+-------------------+
> | full_text| id|
> +--------------------+-------------------+
>
> I want to tokenize each sentence into pairs of (word, id)
>
> for example, having the record : ("Hi, How are you?", id) I want to
> convert the dataframe to:
> hi, id
> how, id
> are, id
> you, id
> ?, id
>
> So I try :
>
> data.rdd.map(lambda data : (data[0], data[1]))\
> .flatMap(lambda row: (word_tokenize(row[0].lower()), row[1])
>
> but it converts the dataframe to:
> [hi, how, are, you, ?]
>
> How can I do the desired transformation?
>