[jira] [Updated] (HIVE-30020) TIMESTAMP WITH LOCAL TIME ZONE columns lose data / fail to read on ORC tables

["Vikram Ahuja (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-30020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vikram Ahuja updated HIVE-30020:
--------------------------------
    Attachment: 0001-HIVE-30020-Fix-TIMESTAMP-WITH-LOCAL-TIME-ZONE-data-l.patch

>  TIMESTAMP WITH LOCAL TIME ZONE columns lose data / fail to read on ORC tables
> ------------------------------------------------------------------------------
>
>                 Key: HIVE-30020
>                 URL: https://issues.apache.org/jira/browse/HIVE-30020
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Vikram Ahuja
>            Assignee: Vikram Ahuja
>            Priority: Major
>         Attachments: 0001-HIVE-30020-Fix-TIMESTAMP-WITH-LOCAL-TIME-ZONE-data-l.patch
>
>
>  TIMESTAMP WITH LOCAL TIME ZONE is not properly supported in ORC format. Though we are able to create and insert data. Select query throws UnsupportedOperationException: Unknown type TIMESTAMP_INSTANT exception. The ORC read/write code (org.apache.hadoop.hive.ql.io.orc) never had a case added for this type when it was introduced, which produces two distinct failures:
>  1. Silent data loss on write. WriterImpl.setColumn() switches on PrimitiveCategory to populate a TimestampColumnVector, but has no case TIMESTAMPLOCALTZ and no default branch so the assignment is silently skipped, leaving the column at its zero-initialized value. INSERT completes with no error, but every row's timestamp is written as epoch 0.
>   2. Read failure. Reading such a column back out of ORC (e.g. via SELECT, using Hive's FetchTask local-read optimization) throws, because:
>   - OrcStruct.createObjectInspector(int columnId, List<OrcProto.Type> types) has no case for TIMESTAMP_INSTANT (the ORC on-disk type for this Hive type) UnsupportedOperationException: Unknown type TIMESTAMP_INSTANT.
>   - RecordReaderImpl.nextValue() has no case for TIMESTAMP_INSTANT → IllegalArgumentException: Unknown type timestamp with local time zone.
>  
>  
>   Steps to reproduce:
> {code:java}
>   CREATE TABLE timestampltz_formats (
>     formatid string,
>     tsval timestamp with local time zone
>   )
>   ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe';
>   LOAD DATA LOCAL INPATH '.../data/files/timestamps_mixed_formats.txt' overwrite into table timestampltz_formats;
>   CREATE TABLE timestampltz_orc_format (
>     formatid string,
>     tsval timestamp with local time zone
>   )
>   stored as orc;
>   insert into timestampltz_orc_format select * from timestampltz_formats;
>   SELECT * FROM timestampltz_orc_format;
> {code}
>  
> Observed: SELECT throws UnsupportedOperationException: Unknown type TIMESTAMP_INSTANT. If the read-side gap is patched in isolation, the exception disappears but every tsval comes back as epoch 0 (e.g. 1969-12-31 16:00:00.0 US/Pacific) instead of the inserted value, because the write silently dropped the data.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)