You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Gerald-G <sh...@gmail.com> on 2015/05/08 05:07:37 UTC

Dismatch when use sparkSQL insert data into a hive table with dynamic partition datetype

Hi:

 Spark version is 1.3.1

 I used sparksql insert data into a hive table with datetype partition

 The column is translate to an int num which is the date diff from
1970-01-01

fat_tbUser is a very simple hive table with partition dtstatdate DateType
And date column userid

  INSERT INTO TABLE fat_tbUser  "
                + "partition(dtstatdate) "
                + "SELECT userid"
                + "'2015-05-08' AS dtstatdate,"
                + "FROM fat_tbUser WHERE
dtStatDate=date_add('2015-05-08',-2)"


When I run it in sparkSQL, HIVE get the   dtstatdate=16563


Any Suggestion?

Thanks
Yours

Meng

Re: Dismatch when use sparkSQL insert data into a hive table with dynamic partition datetype

Posted by Gerald-G <sh...@gmail.com>.

Resolved by create udf to match different format of DateType in sparksql
and HIVE

suggest fix it in next release

On Fri, May 8, 2015 at 11:07 AM, Gerald-G <sh...@gmail.com> wrote:

> Hi:
>
>  Spark version is 1.3.1
>
>  I used sparksql insert data into a hive table with datetype partition
>
>  The column is translate to an int num which is the date diff from
> 1970-01-01
>
> fat_tbUser is a very simple hive table with partition dtstatdate DateType
>   And date column userid
>
>   INSERT INTO TABLE fat_tbUser  "
>                 + "partition(dtstatdate) "
>                 + "SELECT userid"
>                 + "'2015-05-08' AS dtstatdate,"
>                 + "FROM fat_tbUser WHERE
> dtStatDate=date_add('2015-05-08',-2)"
>
>
> When I run it in sparkSQL, HIVE get the   dtstatdate=16563
>
>
> Any Suggestion?
>
> Thanks
> Yours
>
> Meng
>
>