You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Jimmy Xiang <jx...@cloudera.com> on 2015/02/24 21:55:47 UTC
Review Request 31379: HIVE-9772 Hive parquet timestamp conversion
doesn't work with new Parquet
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31379/
-----------------------------------------------------------
Review request for hive and Brock Noland.
Bugs: HIVE-9772
https://issues.apache.org/jira/browse/HIVE-9772
Repository: hive-git
Description
-------
Using Conf to pass timestamp conversion info around, instead of readSupportMetadata, which is not supported by latest Parquet any more
Diffs
-----
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ETypeConverter.java 377e362
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/DataWritableReadSupport.java 47cd682
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/ParquetRecordReaderWrapper.java 6dc85fa
Diff: https://reviews.apache.org/r/31379/diff/
Testing
-------
Unit test, qtest
Thanks,
Jimmy Xiang
Re: Review Request 31379: HIVE-9772 Hive parquet timestamp conversion
doesn't work with new Parquet
Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31379/#review74080
-----------------------------------------------------------
Ship it!
Ship It!
- Brock Noland
On Feb. 24, 2015, 8:55 p.m., Jimmy Xiang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31379/
> -----------------------------------------------------------
>
> (Updated Feb. 24, 2015, 8:55 p.m.)
>
>
> Review request for hive and Brock Noland.
>
>
> Bugs: HIVE-9772
> https://issues.apache.org/jira/browse/HIVE-9772
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Using Conf to pass timestamp conversion info around, instead of readSupportMetadata, which is not supported by latest Parquet any more
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ETypeConverter.java 377e362
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/DataWritableReadSupport.java 47cd682
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/ParquetRecordReaderWrapper.java 6dc85fa
>
> Diff: https://reviews.apache.org/r/31379/diff/
>
>
> Testing
> -------
>
> Unit test, qtest
>
>
> Thanks,
>
> Jimmy Xiang
>
>
Re: Review Request 31379: HIVE-9772 Hive parquet timestamp conversion
doesn't work with new Parquet
Posted by Jimmy Xiang <jx...@cloudera.com>.
> On Feb. 25, 2015, 6:50 p.m., Brock Noland wrote:
> > Hey, can you generate a parquet file with this on and off an ensure that the flag is stored correctly? As other engines like impala use this flag...
This flag is not stored. Parquet file doesn't contains such a flag. This info is derived by the createBy info in a parquet file, and the timestamp conversion setting in Hive. We just use the Conf object to pass such info to the converters.
> On Feb. 25, 2015, 6:50 p.m., Brock Noland wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/DataWritableReadSupport.java, line 157
> > <https://reviews.apache.org/r/31379/diff/1/?file=874391#file874391line157>
> >
> > this populates the metadata written by parquet, correct?
This is for reading from a parquet file. The metadata is used by Hive to decide how to do the configuration. It is not stored in the parquet file.
- Jimmy
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31379/#review73956
-----------------------------------------------------------
On Feb. 24, 2015, 8:55 p.m., Jimmy Xiang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31379/
> -----------------------------------------------------------
>
> (Updated Feb. 24, 2015, 8:55 p.m.)
>
>
> Review request for hive and Brock Noland.
>
>
> Bugs: HIVE-9772
> https://issues.apache.org/jira/browse/HIVE-9772
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Using Conf to pass timestamp conversion info around, instead of readSupportMetadata, which is not supported by latest Parquet any more
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ETypeConverter.java 377e362
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/DataWritableReadSupport.java 47cd682
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/ParquetRecordReaderWrapper.java 6dc85fa
>
> Diff: https://reviews.apache.org/r/31379/diff/
>
>
> Testing
> -------
>
> Unit test, qtest
>
>
> Thanks,
>
> Jimmy Xiang
>
>
Re: Review Request 31379: HIVE-9772 Hive parquet timestamp conversion
doesn't work with new Parquet
Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31379/#review73956
-----------------------------------------------------------
Hey, can you generate a parquet file with this on and off an ensure that the flag is stored correctly? As other engines like impala use this flag...
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/DataWritableReadSupport.java
<https://reviews.apache.org/r/31379/#comment120367>
this populates the metadata written by parquet, correct?
- Brock Noland
On Feb. 24, 2015, 8:55 p.m., Jimmy Xiang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31379/
> -----------------------------------------------------------
>
> (Updated Feb. 24, 2015, 8:55 p.m.)
>
>
> Review request for hive and Brock Noland.
>
>
> Bugs: HIVE-9772
> https://issues.apache.org/jira/browse/HIVE-9772
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Using Conf to pass timestamp conversion info around, instead of readSupportMetadata, which is not supported by latest Parquet any more
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ETypeConverter.java 377e362
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/DataWritableReadSupport.java 47cd682
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/ParquetRecordReaderWrapper.java 6dc85fa
>
> Diff: https://reviews.apache.org/r/31379/diff/
>
>
> Testing
> -------
>
> Unit test, qtest
>
>
> Thanks,
>
> Jimmy Xiang
>
>