You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Sergio Pena <se...@cloudera.com> on 2016/02/02 20:10:14 UTC
Re: Review Request 41821: HIVE-12767: Implement table property to
address Parquet int96 timestamp bug
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41821/
-----------------------------------------------------------
(Updated Feb. 2, 2016, 7:10 p.m.)
Review request for hive, Ryan Blue, Mohammad Islam, Reuben Kuhnert, and Szehon Ho.
Changes
-------
I forgot to add data/files/hive2.0_int96_timestamp-bug.parq
Bugs: HIVE-12767
https://issues.apache.org/jira/browse/HIVE-12767
Repository: hive-git
Description
-------
The following exit criteria is addressed in this patch:
* Hive will read Parquet MR int96 timestamp data and adjust values using a time zone from a table property, if set, or using the local time zone if it is absent. No adjustment will be applied to data written by Impala.
* Hive will write Parquet int96 timestamps using a time zone adjustment from the same table property, if set, or using the local time zone if it is absent. This keeps the data in the table consistent.
* New tables created by Hive will set the table property to UTC if the global option to set the property for new tables is enabled.
* Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set the property unless the global setting to do so is enabled.
* Tables created using CREATE TABLE LIKE <OTHER TABLE> will copy the property of the table that is copied.
To set the timezone table property, use this:
create table tbl1 (ts timestamp) stored as parquet tblproperties ('parquet.mr.int96.write.zone'='PST');
To set UTC as default timezone table property on new tables created, use this:
set parquet.mr.int96.enable.utc.write.zone=true;
create table tbl2 (ts timestamp) stored as parquet;
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java bfd88f82ee86740f60baed122ca050dd542d637c
data/files/hive2.0_int96_timestamp-bug.parq PRE-CREATION
data/files/impala_int96_timestamp.parq PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 2e4591313f38d9bd1bfe1b54968c835fdcd53717
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetOutputFormat.java bfb48a987ce89a373f3da63c9162546c6eda43a9
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ETypeConverter.java ec0dd818f688ab92feb46be4fb6040ede5ac756a
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/DataWritableReadSupport.java 53f3b72b790d87a75a7cd1d77d8f011c29c41188
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/ParquetRecordReaderWrapper.java 74a1a82047613189678716f765bfaa9ac39b7618
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetTableUtils.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/timestamp/NanoTimeUtils.java aace48ee7d145d199163286d21e4ee7694140d6f
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/DataWritableWriteSupport.java f4621e5dbb81e8d58c4572c901ec9d1a7ca8c012
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/DataWritableWriter.java 69272dc41dbc5fe29ab4c98e730b591c28f3a297
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 8c880c39823a9279e32ee191d9f250ea0014888f
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestDataWritableWriter.java 70491390ba2b90f32ef9963be7b19e57672241f3
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/convert/TestETypeConverter.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/serde/TestParquetTimestampUtils.java ec6def5b9ac5f12e6a7cb24c4f4998a6ca6b4a8e
ql/src/test/queries/clientnegative/parquet_int96_timestamp_errors.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_int96_timestamp.q PRE-CREATION
ql/src/test/results/clientnegative/parquet_int96_timestamp_errors.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_int96_timestamp.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/41821/diff/
Testing
-------
Added unit and q-tests:
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestDataWritableWriter.java
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/convert/TestETypeConverter.java
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/serde/TestParquetTimestampUtils.java
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/timestamp/TestParquetTimestampConverter.java
ql/src/test/queries/clientpositive/parquet_int96_timestamp.q
Thanks,
Sergio Pena