You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2019/08/05 09:11:00 UTC
[jira] [Updated] (PARQUET-1627) Update specification so that legacy
timestamp logical types can be written for local semantics as well
[ https://issues.apache.org/jira/browse/PARQUET-1627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated PARQUET-1627:
------------------------------------
Labels: pull-request-available (was: )
> Update specification so that legacy timestamp logical types can be written for local semantics as well
> ------------------------------------------------------------------------------------------------------
>
> Key: PARQUET-1627
> URL: https://issues.apache.org/jira/browse/PARQUET-1627
> Project: Parquet
> Issue Type: Task
> Components: parquet-format
> Reporter: Zoltan Ivanfi
> Assignee: Nandor Kollar
> Priority: Major
> Labels: pull-request-available
>
> The rules for TIMESTAMP forward-compatibility were created based on the assumption that TIMESTAMP_MILLIS and TIMESTAMP_MICROS have only been used in the instant aka. UTC-normalized semantics so far.
> From this false premise it followed that TIMESTAMPs with local semantics were a new type and did not need to be annotated with the old types to maintain compatibility. In fact, annotating them with the old types were considered to be harmful, since it would have mislead older readers into thinking that they can read TIMESTAMPs with local semantics, when in reality they would have misinterpreted them as TIMESTAMPs with instant semantics. This would have lead to a difference of several hours, corresponding to the time zone offset.
> In reality, however, this misinterpretation of timestamps has already been going on for a while, since Arrow annotates local timestamps with TIMESTAMP_MILLIS or TIMESTMAP_MICROS.
> To maintain forward compatibilty of local timestamps, the specification should allow annotating them with the legacy timestamp logical types.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)