[jira] [Work logged] (HIVE-25104) Backward incompatible timestamp serialization in Parquet for certain timezones

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-25104?focusedWorklogId=597577&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-597577 ]

ASF GitHub Bot logged work on HIVE-25104:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 17/May/21 12:07
            Start Date: 17/May/21 12:07
    Worklog Time Spent: 10m 
      Work Description: zabetak opened a new pull request #2282:
URL: https://github.com/apache/hive/pull/2282


   ### What changes were proposed in this pull request?
       
   1. Add new read/write config properties to control legacy zone conversions in Parquet.
   2. Deprecate hive.parquet.timestamp.legacy.conversion.enabled property since it is not clear if it applies on conversion during read or write.
   3. Exploit file metadata and property to choose between new/old conversion rules.
   4. Update existing tests to remove usages of now deprecated hive.parquet.timestamp.legacy.conversion.enabled property.
   5. Simplify NanoTimeUtils#getTimestamp & NanoTimeUtils#getNanoTime by removing 'skipConversion' parameter
   
   ### Why are the changes needed?
   1. Provide the end-users the possibility to write backward compatible timestamps in Parquet files so that files can be read correctly by older versions.
   2. Improve code readability of NanoTimeUtils APIs.
   
   ### Does this PR introduce _any_ user-facing change?
   No.
   
   ### How was this patch tested?
   1. Add timestamp read/write compatibility test with Hive2 Parquet APIs (`TestParquetTimestampsHive2Compatibility`)
   2. Add qtest writing timestamps in Parquet using legacy zone conversions (`parquet_int96_legacy_compatibility_timestamp.q`)
   ```
   mvn test -Dtest=*Timestamp*
   cd itests/qtest
   mvn test -Dtest=TestMiniLlapLocalCliDriver -Dqfile_regex=".*timestamp.*" -Dtest.output.overwrite
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

            Worklog Id:     (was: 597577)
    Remaining Estimate: 0h
            Time Spent: 10m

> Backward incompatible timestamp serialization in Parquet for certain timezones
> ------------------------------------------------------------------------------
>
>                 Key: HIVE-25104
>                 URL: https://issues.apache.org/jira/browse/HIVE-25104
>             Project: Hive
>          Issue Type: Bug
>          Components: Serializers/Deserializers
>    Affects Versions: 3.1.2
>            Reporter: Stamatis Zampetakis
>            Assignee: Stamatis Zampetakis
>            Priority: Major
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> HIVE-12192, HIVE-20007 changed the way that timestamp computations are performed and to some extend how timestamps are serialized and deserialized in files (Parquet, Avro, Orc).
> In versions that include HIVE-12192 or HIVE-20007 the serialization in Parquet files is not backwards compatible. In other words writing timestamps with a version of Hive that includes HIVE-12192/HIVE-20007 and reading them with another (not including the previous issues) may lead to different results depending on the default timezone of the system.
> Consider the following scenario where the default system timezone is set to US/Pacific.
> At apache/master commit 37f13b02dff94e310d77febd60f93d5a205254d3
> {code:sql}
> CREATE EXTERNAL TABLE employee(eid INT,birth timestamp) STORED AS PARQUET
>  LOCATION '/tmp/hiveexttbl/employee';
> INSERT INTO employee VALUES (1, '1880-01-01 00:00:00');
> INSERT INTO employee VALUES (2, '1884-01-01 00:00:00');
> INSERT INTO employee VALUES (3, '1990-01-01 00:00:00');
> SELECT * FROM employee;
> {code}
> |1|1880-01-01 00:00:00|
> |2|1884-01-01 00:00:00|
> |3|1990-01-01 00:00:00|
> At apache/branch-2.3 commit 324f9faf12d4b91a9359391810cb3312c004d356
> {code:sql}
> CREATE EXTERNAL TABLE employee(eid INT,birth timestamp) STORED AS PARQUET
>  LOCATION '/tmp/hiveexttbl/employee';
> SELECT * FROM employee;
> {code}
> |1|1879-12-31 23:52:58|
> |2|1884-01-01 00:00:00|
> |3|1990-01-01 00:00:00|
> The timestamp for {{eid=1}} in branch-2.3 is different from the one in master.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)