You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Soheil Pourbafrani <so...@gmail.com> on 2018/06/19 17:38:20 UTC

How can I do the following simple scenario in spark

Hi, I have a JSON file in the following structure:
+--------------------+-------------------+
|           full_text|                 id|
+--------------------+-------------------+

I want to tokenize each sentence into pairs of (word, id)

for example, having the record : ("Hi, How are you?", id) I want to convert
the dataframe to:
hi, id
how, id
are, id
you, id
?, id

So I try :

data.rdd.map(lambda data : (data[0], data[1]))\
   .flatMap(lambda row: (word_tokenize(row[0].lower()), row[1])

but it converts the dataframe to:
[hi, how, are, you, ?]

How can I do the desired transformation?

Re: How can I do the following simple scenario in spark

Posted by Sonal Goyal <so...@gmail.com>.
Try flatMapToPair instead of flatMap

Thanks,
Sonal
Nube Technologies <http://www.nubetech.co>

<http://in.linkedin.com/in/sonalgoyal>



On Tue, Jun 19, 2018 at 11:08 PM, Soheil Pourbafrani <so...@gmail.com>
wrote:

> Hi, I have a JSON file in the following structure:
> +--------------------+-------------------+
> |           full_text|                 id|
> +--------------------+-------------------+
>
> I want to tokenize each sentence into pairs of (word, id)
>
> for example, having the record : ("Hi, How are you?", id) I want to
> convert the dataframe to:
> hi, id
> how, id
> are, id
> you, id
> ?, id
>
> So I try :
>
> data.rdd.map(lambda data : (data[0], data[1]))\
>    .flatMap(lambda row: (word_tokenize(row[0].lower()), row[1])
>
> but it converts the dataframe to:
> [hi, how, are, you, ?]
>
> How can I do the desired transformation?
>