data type transform when creating an RDD object

["Lin, Hao" <Ha...@finra.org>]

Hi,

Quick question on data type transform when creating RDD object.

I want to create a person object with "name" and DOB(date of birth):
        case class Person(name: String, DOB: java.sql.Date)

then I want to create an RDD from a text file without the header, e.g. "name" and "DOB". I have problem of the following expression, because p(1) is previously defined as java.sql.Date in the case class:

    val people = sc.textFile("examples/src/main/resources/people.txt").map(_.split(",")).map(p => Person(p(0), p(1))).toDF()

36: error: type mismatch;
found   : String
required: java.sql.Date

so how do I transform p(1) in the above expression to java.sql.Date.

any help will be appreciated here.

thanks

Confidentiality Notice::  This email, including attachments, may include non-public, proprietary, confidential or legally privileged information.  If you are not an intended recipient or an authorized agent of an intended recipient, you are hereby notified that any dissemination, distribution or copying of the information contained in or transmitted with this e-mail is unauthorized and strictly prohibited.  If you have received this email in error, please notify the sender by replying to this message and permanently delete this e-mail, its attachments, and any copies of it immediately.  You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.

Re: data type transform when creating an RDD object

[Daniel Siegmann <da...@teamaol.com>]

This should do it (for the implementation of your parse method, Google
should easily provide information - SimpleDateFormatter is probably what
you want):

def parseDate(s: String): java.sql.Date = { ... }
val people = sc.textFile("examples/src/main/resources/people.txt")
               .map(_.split(","))
               .map(p => Person(p(0), *parseDate(*p(1*)*)))
               .toDF()


On Wed, Feb 17, 2016 at 11:47 AM, Lin, Hao <Ha...@finra.org> wrote:

> Hi,
>
>
>
> Quick question on data type transform when creating RDD object.
>
>
>
> I want to create a person object with “name” and DOB(date of birth):
>
>         case class Person(name: String, DOB: java.sql.Date)
>
>
>
> then I want to create an RDD from a text file without the header, e.g.
> “name” and “DOB”. I have problem of the following expression, because p(1)
> is previously defined as java.sql.Date in the case class:
>
>
>
>     val people =
> sc.textFile("examples/src/main/resources/people.txt").map(_.split(",")).map(p
> => Person(p(0), p(1))).toDF()
>
>
>
> 36: error: type mismatch;
>
> found   : String
>
> required: java.sql.Date
>
>
>
> so how do I transform p(1) in the above expression to java.sql.Date.
>
>
>
> any help will be appreciated here.
>
>
>
> thanks
> Confidentiality Notice:: This email, including attachments, may include
> non-public, proprietary, confidential or legally privileged information. If
> you are not an intended recipient or an authorized agent of an intended
> recipient, you are hereby notified that any dissemination, distribution or
> copying of the information contained in or transmitted with this e-mail is
> unauthorized and strictly prohibited. If you have received this email in
> error, please notify the sender by replying to this message and permanently
> delete this e-mail, its attachments, and any copies of it immediately. You
> should not retain, copy or use this e-mail or any attachment for any
> purpose, nor disclose all or any part of the contents to any other person.
> Thank you.
>