You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Krishna <re...@gmail.com> on 2015/12/08 18:47:42 UTC

Merge rows into csv

Hi,

what is the most efficient way to perform a group-by operation in Spark and
merge rows into csv?

Here is the current RDD
-----------------
ID   STATE
-----------------
1       TX
1        NY
1        FL
2        CA
2        OH
-----------------

This is the required output:
-------------------------
ID    CSV_STATE
-------------------------
1     TX,NY,FL
2     CA,OH
-------------------------

Re: Merge rows into csv

Posted by ayan guha <gu...@gmail.com>.

reduceByKey would be a perfect fit for you

On Wed, Dec 9, 2015 at 4:47 AM, Krishna <re...@gmail.com> wrote:

> Hi,
>
> what is the most efficient way to perform a group-by operation in Spark
> and merge rows into csv?
>
> Here is the current RDD
> -----------------
> ID   STATE
> -----------------
> 1       TX
> 1        NY
> 1        FL
> 2        CA
> 2        OH
> -----------------
>
> This is the required output:
> -------------------------
> ID    CSV_STATE
> -------------------------
> 1     TX,NY,FL
> 2     CA,OH
> -------------------------
>



-- 
Best Regards,
Ayan Guha