[jira] [Created] (IMPALA-9290) ORC scanner should support schema evolution between date and timestamp types

["Gabor Kaszab (Jira)" <ji...@apache.org>]

Gabor Kaszab created IMPALA-9290:
------------------------------------

             Summary: ORC scanner should support schema evolution between date and timestamp types
                 Key: IMPALA-9290
                 URL: https://issues.apache.org/jira/browse/IMPALA-9290
             Project: IMPALA
          Issue Type: Bug
          Components: Backend
    Affects Versions: Impala 3.3.0
            Reporter: Gabor Kaszab


*This is the desired use case:*
1. Create an ORC table TBL1 with a DATE column.
2. Create an ORC table TBL2 with a TIMESTAMP column that has the same location as TBL1.
3. Insert some DATE values into TBL1 and some TIMESTAMP values into TBL2.
4. select from TBL1 returns both DATE and TIMESTAMP values (converted to DATE).
5. select from TBL2 returns both DATE and TIMESTAMPS values. The DATE values are converted to TIMESTAMP.

Without this feature Impala return an error:

{code:java}
ERROR: Type mismatch: table column DATE is map to column timestamp in ORC file 'hdfs://localhost:20500/test-warehouse/orc_date_tbl/000000_0_copy_1'
{code}

*Note:*
With https://issues.apache.org/jira/browse/IMPALA-8801 implementing Date type for ORC it is possible to read date values in ORC format. However, writing is still not supported and has to be done by Hive.

*Let me copy-paste a code review comment from IMPALA-8801 as a suggestion for the implementation:*
We can modify OrcTimestampReader to support reading orc::TimestampVectorBatch into Date type slots. In its constructor it knows which kind of slots (timestamp or date) it's writting to. So in ReadValue() it can have different behaviors based on different modes (timestamp values => timestamp slots / timestamp values => date slots). We can do the same on OrcDateColumnReader to let it support reading ORC Date values into Timestamp type slots.

Note that the life cycle of a OrcColumnReader is within the life cycle of the HdfsOrcScanner which only reads a split of an ORC file, and an ORC file can't have two types for one column (e.g. column1 is timestamp in stripe1 and is date in stripe2). So we don't need to deal with different batch types in UpdateInputBatch().

BTW, It'd be better to add test coverage for this type compactibility check in test_scanners.py (See TestOrc.test_type_conversions).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)