You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Nikhil Gupta (Jira)" <ji...@apache.org> on 2021/06/24 14:45:00 UTC
[jira] [Comment Edited] (HIVE-25104) Backward incompatible
timestamp serialization in Parquet for certain timezones
[ https://issues.apache.org/jira/browse/HIVE-25104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17368895#comment-17368895 ]
Nikhil Gupta edited comment on HIVE-25104 at 6/24/21, 2:44 PM:
---------------------------------------------------------------
I have a list of issues which we can club together: (These are after hive 3.1 release):
# HIVE-25268: date_format udf doesn't work for dates prior to 1900 if the timezone is different from UTC
# HIVE-25093: date_format() UDF is returning output in UTC time zone only (Ashish Sharma, reviewed by Adesh Rao, Nikhil Gupta, Sankar Hariappan)
# HIVE-25104: Backward incompatible timestamp serialization in Parquet for certain timezones (Stamatis Zampetakis, reviewed by Jesus Camacho Rodriguez)
# HIVE-24113: NPE in GenericUDFToUnixTimeStamp (#1460) (Raj Kumar Singh, reviewed by Zoltan Haindrich and Laszlo Pinter)
# HIVE-24074: Incorrect handling of timestamp in Parquet/Avro when written in certain time zones in versions before Hive 3.x (Jesus Camacho Rodriguez, reviewed by Prasanth Jayachandran)
# HIVE-22840: Race condition in formatters of TimestampColumnVector and DateColumnVector (Shubham Chaurasia, reviewed by Jesus Camacho Rodriguez)
# HIVE-22589: Add storage support for ProlepticCalendar in ORC, Parquet, and Avro (Jesus Camacho Rodriguez, reviewed by David Lavati, L<C3><A1>szl<C3><B3> Bodor, Prasanth Jayachandran)
# HIVE-22405: Add ColumnVector support for ProlepticCalendar (L<C3><A1>szl<C3><B3> Bodor via Owen O'Malley, Jesus Camacho Rodriguez)
# HIVE-22331: unix_timestamp without argument returns timestamp in millisecond instead of second (Naresh P R, reviewed Jesus Camacho Rodriguez)
# HIVE-22170: from_unixtime and unix_timestamp should use user session time zone (Jesus Camacho Rodriguez, reviewed by Vineet Garg)
# HIVE-21729: Arrow serializer sometimes shifts timestamp by one second (Shubham Chaurasia, reviewed by Sankar Hariappan)
# HIVE-21291: Restore historical way of handling timestamps in Avro while keeping the new semantics at the same time (Karen Coppage, reviewed by Jesus Camacho Rodriguez)
was (Author: gupta.nikhil0007):
I have a list of issues which we can club together: (These are after hive 3.1 release):
# HIVE-25093: date_format() UDF is returning output in UTC time zone only (Ashish Sharma, reviewed by Adesh Rao, Nikhil Gupta, Sankar Hariappan)
# HIVE-25104: Backward incompatible timestamp serialization in Parquet for certain timezones (Stamatis Zampetakis, reviewed by Jesus Camacho Rodriguez)
# HIVE-24113: NPE in GenericUDFToUnixTimeStamp (#1460) (Raj Kumar Singh, reviewed by Zoltan Haindrich and Laszlo Pinter)
# HIVE-24074: Incorrect handling of timestamp in Parquet/Avro when written in certain time zones in versions before Hive 3.x (Jesus Camacho Rodriguez, reviewed by Prasanth Jayachandran)
# HIVE-22840: Race condition in formatters of TimestampColumnVector and DateColumnVector (Shubham Chaurasia, reviewed by Jesus Camacho Rodriguez)
# HIVE-22589: Add storage support for ProlepticCalendar in ORC, Parquet, and Avro (Jesus Camacho Rodriguez, reviewed by David Lavati, L<C3><A1>szl<C3><B3> Bodor, Prasanth Jayachandran)
# HIVE-22405: Add ColumnVector support for ProlepticCalendar (L<C3><A1>szl<C3><B3> Bodor via Owen O'Malley, Jesus Camacho Rodriguez)
# HIVE-22331: unix_timestamp without argument returns timestamp in millisecond instead of second (Naresh P R, reviewed Jesus Camacho Rodriguez)
# HIVE-22170: from_unixtime and unix_timestamp should use user session time zone (Jesus Camacho Rodriguez, reviewed by Vineet Garg)
# HIVE-21729: Arrow serializer sometimes shifts timestamp by one second (Shubham Chaurasia, reviewed by Sankar Hariappan)
# HIVE-21291: Restore historical way of handling timestamps in Avro while keeping the new semantics at the same time (Karen Coppage, reviewed by Jesus Camacho Rodriguez)
> Backward incompatible timestamp serialization in Parquet for certain timezones
> ------------------------------------------------------------------------------
>
> Key: HIVE-25104
> URL: https://issues.apache.org/jira/browse/HIVE-25104
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 3.1.0
> Reporter: Stamatis Zampetakis
> Assignee: Stamatis Zampetakis
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> HIVE-12192, HIVE-20007 changed the way that timestamp computations are performed and to some extend how timestamps are serialized and deserialized in files (Parquet, Avro).
> In versions that include HIVE-12192 or HIVE-20007 the serialization in Parquet files is not backwards compatible. In other words writing timestamps with a version of Hive that includes HIVE-12192/HIVE-20007 and reading them with another (not including the previous issues) may lead to different results depending on the default timezone of the system.
> Consider the following scenario where the default system timezone is set to US/Pacific.
> At apache/master commit 37f13b02dff94e310d77febd60f93d5a205254d3
> {code:sql}
> CREATE EXTERNAL TABLE employee(eid INT,birth timestamp) STORED AS PARQUET
> LOCATION '/tmp/hiveexttbl/employee';
> INSERT INTO employee VALUES (1, '1880-01-01 00:00:00');
> INSERT INTO employee VALUES (2, '1884-01-01 00:00:00');
> INSERT INTO employee VALUES (3, '1990-01-01 00:00:00');
> SELECT * FROM employee;
> {code}
> |1|1880-01-01 00:00:00|
> |2|1884-01-01 00:00:00|
> |3|1990-01-01 00:00:00|
> At apache/branch-2.3 commit 324f9faf12d4b91a9359391810cb3312c004d356
> {code:sql}
> CREATE EXTERNAL TABLE employee(eid INT,birth timestamp) STORED AS PARQUET
> LOCATION '/tmp/hiveexttbl/employee';
> SELECT * FROM employee;
> {code}
> |1|1879-12-31 23:52:58|
> |2|1884-01-01 00:00:00|
> |3|1990-01-01 00:00:00|
> The timestamp for {{eid=1}} in branch-2.3 is different from the one in master.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)