You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Brock Noland <br...@cloudera.com> on 2014/11/02 19:30:30 UTC
Re: Review Request 27404: HIVE-8359 Map containing null values are not
correctly written in Parquet files
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27404/#review59527
-----------------------------------------------------------
Hi,
Nice work! Have you verified that some of the other parquet tools can read the data generated by this patch? ex. Impala or MapReduce?
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/DataWritableWriter.java
<https://reviews.apache.org/r/27404/#comment100799>
Please remove this import change. We typically don't want to do wildcard imports. I am guessing this was done by your IDE
- Brock Noland
On Oct. 30, 2014, 11:43 p.m., Sergio Pena wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27404/
> -----------------------------------------------------------
>
> (Updated Oct. 30, 2014, 11:43 p.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-8359
> https://issues.apache.org/jira/browse/HIVE-8359
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> The patch changes the way DataWritableWriter class writes an array of elements to the Parquet record.
> It wraps each array record into a new startGroup/endGroup block so that Parquet can detect null values on those optional fields.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/DataWritableWriter.java c7078efe27482df0a11dd68ac068da27dbcf51b3
> ql/src/test/queries/clientpositive/parquet_map_null.q PRE-CREATION
> ql/src/test/results/clientpositive/parquet_map_null.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/27404/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergio Pena
>
>
Re: Review Request 27404: HIVE-8359 Map containing null values are not
correctly written in Parquet files
Posted by Sergio Pena <se...@cloudera.com>.
> On Nov. 2, 2014, 6:30 p.m., Brock Noland wrote:
> > Hi,
> >
> > Nice work! Have you verified that some of the other parquet tools can read the data generated by this patch? ex. Impala or MapReduce?
I run some tests on Impala, but it does not support complex data types, such as maps or arrays.
See http://www.cloudera.com/content/cloudera/en/documentation/cloudera-impala/latest/topics/impala_datatypes.html
Also, see this error from a query to a parquet with map data types:
ERROR: AnalysisException: Expr 'default.parquet_1.avreau_col_1' in select list of root statement returns a complex type 'MAP<STRING,STRING>'.
Only scalar types are allowed in the select list of the root statement.
- Sergio
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27404/#review59527
-----------------------------------------------------------
On Oct. 30, 2014, 11:43 p.m., Sergio Pena wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27404/
> -----------------------------------------------------------
>
> (Updated Oct. 30, 2014, 11:43 p.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-8359
> https://issues.apache.org/jira/browse/HIVE-8359
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> The patch changes the way DataWritableWriter class writes an array of elements to the Parquet record.
> It wraps each array record into a new startGroup/endGroup block so that Parquet can detect null values on those optional fields.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/DataWritableWriter.java c7078efe27482df0a11dd68ac068da27dbcf51b3
> ql/src/test/queries/clientpositive/parquet_map_null.q PRE-CREATION
> ql/src/test/results/clientpositive/parquet_map_null.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/27404/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergio Pena
>
>