You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "szingerpeter (via GitHub)" <gi...@apache.org> on 2023/01/21 19:08:16 UTC

[GitHub] [hudi] szingerpeter opened a new issue, #7724: failed to read timestamp from hive

szingerpeter opened a new issue, #7724:
URL: https://github.com/apache/hudi/issues/7724

   Hy,
   
   I'm using Hudi CLI version 1.0; hudi version 0.11.0; Spark version 3.2.1-amzn-0 and Hive version 3.1.3-amzn-0.
   
   the error i'm getting:
   
   ```
    java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be cast to org.apache.hadoop.hive.serde2.io.TimestampWritable
           at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableTimestampObjectInspector.getPrimitiveJavaObject(WritableTimestampObjectInspector.java:39)
   ```
   
   write settings:
   ```
   hoodie.datasource.hive_sync.support_timestamp: True
   ```
   
   spark settings:
   ```
   spark.serializer: org.apache.spark.serializer.KryoSerializer
   spark.sql.hive.convertMetastoreParquet: false
   hive.input.format: org.apache.hadoop.hive.ql.io.HiveInputFormat
   spark.sql.avro.datetimeRebaseModeInWrite: CORRECTED
   ```
   i've seen multiple other issues reporting the same error and PRs, but couldn't figure out what would be the right solution
   [#2509](https://github.com/apache/hudi/issues/2509)
   [#2544 ](https://github.com/apache/hudi/issues/2544)
   [#2869](https://github.com/apache/hudi/issues/2869)
   [#3391](https://github.com/apache/hudi/pull/3391)
   [#3378](https://github.com/apache/hudi/pull/3378)
   [#6232](https://github.com/apache/hudi/issues/6232)
   
   also, [this link](https://hudi.apache.org/docs/configurations.html#HIVE_SUPPORT_TIMESTAMP) on the [FAQ page](https://cwiki.apache.org/confluence/display/HUDI/FAQ) doesn't work


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on issue #7724: [SUPPORT] failed to read timestamp from hive

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #7724:
URL: https://github.com/apache/hudi/issues/7724#issuecomment-1401386744

   What is your timestamp precision, only micro-seconds with precision 6 is synced with Timestamp type in Hive.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] szingerpeter commented on issue #7724: [SUPPORT] failed to read timestamp from hive

Posted by "szingerpeter (via GitHub)" <gi...@apache.org>.
szingerpeter commented on issue #7724:
URL: https://github.com/apache/hudi/issues/7724#issuecomment-1402637185

   update:
   
   if i don't specify the table in advance in hive, but let hudi sync and create a new table it works with artificially adding the `.000001`.
   
   I know it's becoming a bit of a spark-specific question, but do you have an idea on how to cast to timestamp with MS precision with 0 MS?
   ```
   spark.createDataFrame([['2022-01-01 01:01:01.000000']]).withColumn('test', F.col('_1').cast('timestamp')).show(10, False)
   
   +--------------------------+-------------------+
   |_1                        |test               |
   +--------------------------+-------------------+
   |2022-01-01 01:01:01.000000|2022-01-01 01:01:01|
   +--------------------------+-------------------+
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] szingerpeter commented on issue #7724: [SUPPORT] failed to read timestamp from hive

Posted by "szingerpeter (via GitHub)" <gi...@apache.org>.
szingerpeter commented on issue #7724:
URL: https://github.com/apache/hudi/issues/7724#issuecomment-1403478615

   it's not supported.
   
   ```
   pyspark.sql.utils.ParseException:
   DataType timestamp(0) is not supported.(line 1, pos 11)
   
   == SQL ==
   CAST(_1 AS TIMESTAMP(0))
   -----------^^^
   ```
   
   ```
   spark.createDataFrame([['2022-01-01 01:01:01.000000']]).withColumn('test', F.expr('CAST(_1 AS TIMESTAMP(0))'))
   ```
   
   I guess for hudi to save the column as of type `timestamp`, the dataframe must have a timestamp type as well rather than a timestamp formatted string, right?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] szingerpeter commented on issue #7724: failed to read timestamp from hive

Posted by "szingerpeter (via GitHub)" <gi...@apache.org>.
szingerpeter commented on issue #7724:
URL: https://github.com/apache/hudi/issues/7724#issuecomment-1400435894

   i tried setting
   ```
   spark.sql.avro.datetimeRebaseModeInWrite=LEGACY
   ```
   
   as well, but i get the same error:
   ```
   Caused by: java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be cast to org.apache.hadoop.hive.serde2.io.TimestampWritable
           at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableTimestampObjectInspector.getPrimitiveJavaObject(WritableTimestampObjectInspector.java:39)
   ```
   
   the reason i need to set `spark.sql.avro.datetimeRebaseModeInWrite` is, because i was getting the following error:
   ```
   org.apache.spark.SparkUpgradeException: You may get a different result due to the upgrading of Spark 3.0: 
   writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z into Avro
   files can be dangerous, as the files may be read by Spark 2.x or legacy versions of Hive
   later, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
   Gregorian calendar. See more details in SPARK-31404. You can set spark.sql.avro.datetimeRebaseModeInWrite to 'LEGACY' to
   rebase the datetime values w.r.t. the calendar difference during writing, to get maximum
   interoperability. Or set spark.sql.avro.datetimeRebaseModeInWrite to 'CORRECTED' to write the datetime values as it is,
   if you are 100% sure that the written files will only be read by Spark 3.0+ or other
   systems that use Proleptic Gregorian calendar.
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] szingerpeter commented on issue #7724: [SUPPORT] failed to read timestamp from hive

Posted by "szingerpeter (via GitHub)" <gi...@apache.org>.
szingerpeter commented on issue #7724:
URL: https://github.com/apache/hudi/issues/7724#issuecomment-1402619643

   @danny0405 , I tried with artificially creating micro-second precision timestamps (added the suffix `.000001` to each value and casted it to timestamp). saving them results in exactly the same error


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] failed to read timestamp from hive [hudi]

Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go closed issue #7724: [SUPPORT] failed to read timestamp from hive
URL: https://github.com/apache/hudi/issues/7724


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] szingerpeter commented on issue #7724: [SUPPORT] failed to read timestamp from hive

Posted by "szingerpeter (via GitHub)" <gi...@apache.org>.
szingerpeter commented on issue #7724:
URL: https://github.com/apache/hudi/issues/7724#issuecomment-1401518021

   it's got second-precision


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] xicm commented on issue #7724: [SUPPORT] failed to read timestamp from hive

Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on issue #7724:
URL: https://github.com/apache/hudi/issues/7724#issuecomment-1404876542

   #3391 can solve the problem, I tested, and I solved the conflicts on master branch in #7173, you can porting this pr and have a test.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] szingerpeter commented on issue #7724: [SUPPORT] failed to read timestamp from hive

Posted by "szingerpeter (via GitHub)" <gi...@apache.org>.
szingerpeter commented on issue #7724:
URL: https://github.com/apache/hudi/issues/7724#issuecomment-1401651877

   i get the same error even if i don't set `spark.sql.avro.datetimeRebaseModeInWrite`.
   
   Note:
   
   `spark.read.format('hudi').load(PATH)` works fine `spark.sql('select * from TABLE')` fails (and also queries in hive)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] szingerpeter commented on issue #7724: [SUPPORT] failed to read timestamp from hive

Posted by "szingerpeter (via GitHub)" <gi...@apache.org>.
szingerpeter commented on issue #7724:
URL: https://github.com/apache/hudi/issues/7724#issuecomment-1404713245

   well, the issue still persists if i try to query the table from hive (beeline).
   ```
   Error: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be cast to org.apache.hadoop.hive.serde2.io.TimestampWritableV2 (state=,code=0)
   ```
   
   any suggestion to solve this part of the problem?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on issue #7724: [SUPPORT] failed to read timestamp from hive

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #7724:
URL: https://github.com/apache/hudi/issues/7724#issuecomment-1404609544

   It's used for the spark sql schema inference, glad to hear that you solved the problem, so we can close this issue now?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on issue #7724: [SUPPORT] failed to read timestamp from hive

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #7724:
URL: https://github.com/apache/hudi/issues/7724#issuecomment-1403249301

   not sure but explicit cast expr like `cast(f0 as timestamp(0))` work here?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] szingerpeter commented on issue #7724: [SUPPORT] failed to read timestamp from hive

Posted by "szingerpeter (via GitHub)" <gi...@apache.org>.
szingerpeter commented on issue #7724:
URL: https://github.com/apache/hudi/issues/7724#issuecomment-1403894443

   in the end, setting `spark.sql.parquet.outputTimestampType: TIMESTAMP_MICROS` solved the problem.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org