[jira] [Closed] (HUDI-3125) Spark SQL writing timestamp type don't need to disable `spark.sql.datetime.java8API.enabled` manually

["Raymond Xu (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Raymond Xu closed HUDI-3125.
----------------------------
    Resolution: Fixed

> Spark SQL writing timestamp type don't need to disable `spark.sql.datetime.java8API.enabled` manually
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HUDI-3125
>                 URL: https://issues.apache.org/jira/browse/HUDI-3125
>             Project: Apache Hudi
>          Issue Type: Improvement
>          Components: Spark Integration
>            Reporter: Yann Byron
>            Assignee: Yann Byron
>            Priority: Major
>              Labels: pull-request-available, sev:critical, user-support-issues
>             Fix For: 0.10.1
>
>
> {code:java}
> create table h0_p(id int, name string, price double, dt timestamp) using hudi partitioned by(dt) options(type = 'cow', primaryKey = 'id');
> insert into h0_p values (3, 'a1', 10, cast('2021-05-08 00:00:00' as timestamp)); {code}
> By default, that run the sql above will throw exception:
> {code:java}
> Caused by: java.lang.ClassCastException: java.time.Instant cannot be cast to java.sql.Timestamp
>     at org.apache.hudi.AvroConversionHelper$.$anonfun$createConverterToAvro$8(AvroConversionHelper.scala:306)
>     at org.apache.hudi.AvroConversionHelper$.$anonfun$createConverterToAvro$8$adapted(AvroConversionHelper.scala:306)
>     at scala.Option.map(Option.scala:230) {code}
> We need disable `spark.sql.datetime.java8API.enabled` manually to make it work:
> {code:java}
> set spark.sql.datetime.java8API.enabled=false; {code}
> And the command must be executed in the runtime. It can't work if provide this by spark-sql command: `spark-sql --conf spark.sql.datetime.java8API.enabled=false`. That's because this config is forced to enable when launch spark-sql.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)