Manipulating columns in CSV file or Transpose of Array[Array[String]] RDD

["Hingorani, Vineet" <vi...@sap.com>]

Hello all,

Could someone help me with the manipulation of csv file data. I have 'semicolon' separated csv data including doubles and strings. I want to calculate the maximum/average of a column. When I read the file using sc.textFile(test.csv).map(_.split(";"), each field is read as string. Could someone help me with the above manipulation and how to do that.

Or maybe if there is some way to take the transpose of the data and then manipulating the rows in some way?

Thank you in advance, I am struggling with this thing for quite sometime

Regards,
Vineet

Re: Manipulating columns in CSV file or Transpose of Array[Array[String]] RDD

[Victor Tso-Guillen <vt...@paxata.com>]

Assuming the CSV is well-formed (every row has the same number of columns)
and every column is a number, this is how you can do it. You can adjust so
that you pick just the columns you want, of course, by mapping each row to
a new Array that contains just the column values you want. Just be sure the
logic selects the same columns for every row or your stats might look funny.

    val rdd: RDD[Array[Double]] = ???
    rdd.mapPartitions(vs => {
      Iterator(vs.toArray.transpose.map(StatCounter(_)))
    }).reduce((as, bs) => as.zipWithIndex.map {
      case (a, i) => a.merge(bs(i))
    })



On Mon, Aug 25, 2014 at 9:50 AM, Hingorani, Vineet <vineet.hingorani@sap.com
> wrote:

>  Hello Victor,
>
>
>
> I want to do it on multiple columns. I was able to do it on one column by
> the help of Sean using code below.
>
>
>
> val matData = file.map(_.split(";"))
>
> val stats = matData.map(_(2).toDouble).stats()
>
> stats.mean
>
> stats.max
>
>
>
> Thank you
>
>
>
> Vineet
>
>
>
> *From:* Victor Tso-Guillen [mailto:vtso@paxata.com]
> *Sent:* Montag, 25. August 2014 18:34
> *To:* Hingorani, Vineet
> *Cc:* user@spark.apache.org
> *Subject:* Re: Manipulating columns in CSV file or Transpose of
> Array[Array[String]] RDD
>
>
>
> Do you want to do this on one column or all numeric columns?
>
>
>
> On Mon, Aug 25, 2014 at 7:09 AM, Hingorani, Vineet <
> vineet.hingorani@sap.com> wrote:
>
> Hello all,
>
> Could someone help me with the manipulation of csv file data. I have
> 'semicolon' separated csv data including doubles and strings. I want to
> calculate the maximum/average of a column. When I read the file using
> sc.textFile(test.csv).map(_.split(";"), each field is read as string. Could
> someone help me with the above manipulation and how to do that.
>
> Or maybe if there is some way to take the transpose of the data and then
> manipulating the rows in some way?
>
> Thank you in advance, I am struggling with this thing for quite sometime
>
> Regards,
> Vineet
>
>
>

RE: Manipulating columns in CSV file or Transpose of Array[Array[String]] RDD

["Hingorani, Vineet" <vi...@sap.com>]

Hello Victor,

I want to do it on multiple columns. I was able to do it on one column by the help of Sean using code below.


val matData = file.map(_.split(";"))

val stats = matData.map(_(2).toDouble).stats()

stats.mean
stats.max

Thank you

Vineet

From: Victor Tso-Guillen [mailto:vtso@paxata.com]
Sent: Montag, 25. August 2014 18:34
To: Hingorani, Vineet
Cc: user@spark.apache.org
Subject: Re: Manipulating columns in CSV file or Transpose of Array[Array[String]] RDD

Do you want to do this on one column or all numeric columns?

On Mon, Aug 25, 2014 at 7:09 AM, Hingorani, Vineet <vi...@sap.com>> wrote:

Hello all,

Could someone help me with the manipulation of csv file data. I have 'semicolon' separated csv data including doubles and strings. I want to calculate the maximum/average of a column. When I read the file using sc.textFile(test.csv).map(_.split(";"), each field is read as string. Could someone help me with the above manipulation and how to do that.

Or maybe if there is some way to take the transpose of the data and then manipulating the rows in some way?

Thank you in advance, I am struggling with this thing for quite sometime

Regards,
Vineet

Re: Manipulating columns in CSV file or Transpose of Array[Array[String]] RDD

[Victor Tso-Guillen <vt...@paxata.com>]

Do you want to do this on one column or all numeric columns?


On Mon, Aug 25, 2014 at 7:09 AM, Hingorani, Vineet <vineet.hingorani@sap.com
> wrote:

>  Hello all,
>
> Could someone help me with the manipulation of csv file data. I have
> 'semicolon' separated csv data including doubles and strings. I want to
> calculate the maximum/average of a column. When I read the file using
> sc.textFile(test.csv).map(_.split(";"), each field is read as string. Could
> someone help me with the above manipulation and how to do that.
>
> Or maybe if there is some way to take the transpose of the data and then
> manipulating the rows in some way?
>
> Thank you in advance, I am struggling with this thing for quite sometime
>
> Regards,
> Vineet
>