converting timestamp column to a java.util.Date

[Marco Mistroni <mm...@gmail.com>]

HI all
  i am trying to convert a  string column, in a Dataframe , to a
java.util.Date but i am getting this exception

[dispatcher-event-loop-0] INFO org.apache.spark.storage.BlockManagerInfo -
Removed broadcast_0_piece0 on 169.254.2.140:53468 in memory (size: 14.3 KB,
free: 767.4 MB)
Exception in thread "main" java.lang.UnsupportedOperationException: Schema
for type java.util.Date is not supported

here's my code

val tstampToDateFunc:(java.sql.Timestamp => java.util.Date) = ts => new
java.util.Date(ts.getTime)
val tsampConversionFunc = udf(tstampToDateFunc)

sharesDf.withColumn("price", col("_c2").cast("double"))
                            .withColumn("creationTime",
tsampConversionFunc(col("_c1")))

Are there any workarounds?
i m trying to import data into mongoDB via Spark. The source is a csv file
where
i have  1 timestamp column and a bunch of strings..... i will need to
convert that
to something compatible with a mongo's ISODate

kr
 marco

Re: converting timestamp column to a java.util.Date

[Takeshi Yamamuro <li...@gmail.com>]

Hi,

I think Spark UDF can only handle `java.sql.Date`.
So, you need to change the return type in you UDF.

// maropu

On Tue, Jan 24, 2017 at 8:18 AM, Marco Mistroni <mm...@gmail.com> wrote:

> HI all
>   i am trying to convert a  string column, in a Dataframe , to a
> java.util.Date but i am getting this exception
>
> [dispatcher-event-loop-0] INFO org.apache.spark.storage.BlockManagerInfo
> - Removed broadcast_0_piece0 on 169.254.2.140:53468 in memory (size: 14.3
> KB, free: 767.4 MB)
> Exception in thread "main" java.lang.UnsupportedOperationException:
> Schema for type java.util.Date is not supported
>
> here's my code
>
> val tstampToDateFunc:(java.sql.Timestamp => java.util.Date) = ts => new
> java.util.Date(ts.getTime)
> val tsampConversionFunc = udf(tstampToDateFunc)
>
> sharesDf.withColumn("price", col("_c2").cast("double"))
>                             .withColumn("creationTime",
> tsampConversionFunc(col("_c1")))
>
> Are there any workarounds?
> i m trying to import data into mongoDB via Spark. The source is a csv file
> where
> i have  1 timestamp column and a bunch of strings..... i will need to
> convert that
> to something compatible with a mongo's ISODate
>
> kr
>  marco
>
>



-- 
---
Takeshi Yamamuro