You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Ashish Singh <as...@cloudera.com> on 2014/11/23 02:34:31 UTC
Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde to
create table schema from a parquet file
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/
-----------------------------------------------------------
Review request for hive.
Bugs: HIVE-8950
https://issues.apache.org/jira/browse/HIVE-8950
Repository: hive-git
Description
-------
HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
Diffs
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784
data/files/data.parq PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ColInfoFromParquetFile.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd
ql/src/test/queries/clientpositive/parquet_create_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_create_gen_schema1.q PRE-CREATION
ql/src/test/results/clientpositive/parquet_create_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_create_gen_schema1.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/28372/diff/
Testing
-------
Tested by adding appropriate qTests.
Thanks,
Ashish Singh
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde
to create table schema from a parquet file
Posted by Ashish Singh <as...@cloudera.com>.
> On Nov. 24, 2014, 10:44 p.m., Sergio Pena wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java, line 107
> > <https://reviews.apache.org/r/28372/diff/1/?file=773792#file773792line107>
> >
> > Avro uses avro.schema.url to specify the file that will be used for the schema. What if we use the same Avro synatix to specify the parquet schema?
> >
> > Something like: parquet.schema.url
The reason I did not call it "parquet.schema.url", as unlike avro schema file which contains only avro schema, parquet file will contain data as well. It might be misleading for users as there is no concept of parquet schema file. However, advantage of using "parquet.schema.url" will be that usage becomes uniform between avro and parquet. Let me know what you think based on the discussion above.
- Ashish
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/#review62889
-----------------------------------------------------------
On Nov. 27, 2014, 1:08 a.m., Ashish Singh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28372/
> -----------------------------------------------------------
>
> (Updated Nov. 27, 2014, 1:08 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-8950
> https://issues.apache.org/jira/browse/HIVE-8950
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ColInfoFromParquetFile.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd
> ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/28372/diff/
>
>
> Testing
> -------
>
> Tested by adding appropriate qTests.
>
>
> Thanks,
>
> Ashish Singh
>
>
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde
to create table schema from a parquet file
Posted by Sergio Pena <se...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/#review62889
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java
<https://reviews.apache.org/r/28372/#comment105017>
Avro uses avro.schema.url to specify the file that will be used for the schema. What if we use the same Avro synatix to specify the parquet schema?
Something like: parquet.schema.url
- Sergio Pena
On Nov. 23, 2014, 1:34 a.m., Ashish Singh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28372/
> -----------------------------------------------------------
>
> (Updated Nov. 23, 2014, 1:34 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-8950
> https://issues.apache.org/jira/browse/HIVE-8950
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784
> data/files/data.parq PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ColInfoFromParquetFile.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd
> ql/src/test/queries/clientpositive/parquet_create_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_create_gen_schema1.q PRE-CREATION
> ql/src/test/results/clientpositive/parquet_create_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_create_gen_schema1.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/28372/diff/
>
>
> Testing
> -------
>
> Tested by adding appropriate qTests.
>
>
> Thanks,
>
> Ashish Singh
>
>
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde
to create table schema from a parquet file
Posted by Ashish Singh <as...@cloudera.com>.
> On Nov. 25, 2014, 12:37 a.m., Sergio Pena wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ColInfoFromParquetFile.java, line 35
> > <https://reviews.apache.org/r/28372/diff/1/?file=773791#file773791line35>
> >
> > This class will need more work in order to detect unannotated types as specified in the following tickets:
> > https://issues.apache.org/jira/browse/HIVE-8909
> > https://github.com/rdblue/incubator-parquet-format/blob/PARQUET-113-add-list-and-map-spec/LogicalTypes.md
> >
> > I was going to add more comments, but I then noticed that this code will look as little similar (for loops, and converters) to the HIVE-8909 patch. Of course, the HIVE-8909 patch returns converter objects, and this returns the column names & types. So, I was thinking if we can make use of the DataWritableRecordConverter.java class to get the correct converters, and then translate the converters to column names & types.
> >
> > This is what I found while debugging valid parquet files:
> > Each example has 3 blocks:
> > - parquet file schema
> > - hive columne names & types
> > - converters returned by DataWritableRecordConverter
> >
> > Could we use the converter objects and translate them to names?
> >
> > message SingleFieldGroupInList {
> > optional group single_element_groups (LIST) {
> > repeated group single_element_group {
> > required int64 count;
> > }
> > }
> > }
> >
> > single_element_groups ARRAY<BIGINT>
> >
> > hivestructconverter
> > hivecollectionconverter:elementconverter array<>
> > EINT64_CONVERTER bigint
> >
> > --------------------------------------------------------------------------------
> >
> > message HiveRequiredGroupInList {
> > optional group locations (LIST) {
> > repeated group bag {
> > required group element {
> > required double latitude;
> > required double longitude;
> > }
> > }
> > }
> > }
> >
> > locations ARRAY<STRUCT<latitude: DOUBLE, longitude: DOUBLE>>
> >
> > hivestructconverter
> > hivecollectionconverter:elementconverter array<>
> > hivestructconverter struct<>
> > DOUBLE_CONVERTER double
> > DOUBLE_CONVERTER double
> >
> > --------------------------------------------------------------------------------
> >
> > message UnannotatedListOfPrimitives {
> > repeated int32 list_of_ints;
> > }
> >
> > list_of_ints ARRAY<INT>
> >
> > hivestructconverter
> > RepeatedPrimitiveConverter array<>
> > EINT32_CONVERTER int
> >
> > --------------------------------------------------------------------------------
Sergio, thanks for the review and valuable comments!
DataWritableRecordConverter expects hive schema. So, not sure if that can be used to create hive schema. I have updated the patch to take care of rules listed on "https://github.com/rdblue/incubator-parquet-format/blob/PARQUET-113-add-list-and-map-spec/LogicalTypes.md". Kindly take a look.
- Ashish
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/#review62911
-----------------------------------------------------------
On Nov. 27, 2014, 1:08 a.m., Ashish Singh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28372/
> -----------------------------------------------------------
>
> (Updated Nov. 27, 2014, 1:08 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-8950
> https://issues.apache.org/jira/browse/HIVE-8950
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ColInfoFromParquetFile.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd
> ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/28372/diff/
>
>
> Testing
> -------
>
> Tested by adding appropriate qTests.
>
>
> Thanks,
>
> Ashish Singh
>
>
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde
to create table schema from a parquet file
Posted by Sergio Pena <se...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/#review62911
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ColInfoFromParquetFile.java
<https://reviews.apache.org/r/28372/#comment105080>
This class will need more work in order to detect unannotated types as specified in the following tickets:
https://issues.apache.org/jira/browse/HIVE-8909
https://github.com/rdblue/incubator-parquet-format/blob/PARQUET-113-add-list-and-map-spec/LogicalTypes.md
I was going to add more comments, but I then noticed that this code will look as little similar (for loops, and converters) to the HIVE-8909 patch. Of course, the HIVE-8909 patch returns converter objects, and this returns the column names & types. So, I was thinking if we can make use of the DataWritableRecordConverter.java class to get the correct converters, and then translate the converters to column names & types.
This is what I found while debugging valid parquet files:
Each example has 3 blocks:
- parquet file schema
- hive columne names & types
- converters returned by DataWritableRecordConverter
Could we use the converter objects and translate them to names?
message SingleFieldGroupInList {
optional group single_element_groups (LIST) {
repeated group single_element_group {
required int64 count;
}
}
}
single_element_groups ARRAY<BIGINT>
hivestructconverter
hivecollectionconverter:elementconverter array<>
EINT64_CONVERTER bigint
--------------------------------------------------------------------------------
message HiveRequiredGroupInList {
optional group locations (LIST) {
repeated group bag {
required group element {
required double latitude;
required double longitude;
}
}
}
}
locations ARRAY<STRUCT<latitude: DOUBLE, longitude: DOUBLE>>
hivestructconverter
hivecollectionconverter:elementconverter array<>
hivestructconverter struct<>
DOUBLE_CONVERTER double
DOUBLE_CONVERTER double
--------------------------------------------------------------------------------
message UnannotatedListOfPrimitives {
repeated int32 list_of_ints;
}
list_of_ints ARRAY<INT>
hivestructconverter
RepeatedPrimitiveConverter array<>
EINT32_CONVERTER int
--------------------------------------------------------------------------------
- Sergio Pena
On Nov. 23, 2014, 1:34 a.m., Ashish Singh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28372/
> -----------------------------------------------------------
>
> (Updated Nov. 23, 2014, 1:34 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-8950
> https://issues.apache.org/jira/browse/HIVE-8950
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784
> data/files/data.parq PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ColInfoFromParquetFile.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd
> ql/src/test/queries/clientpositive/parquet_create_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_create_gen_schema1.q PRE-CREATION
> ql/src/test/results/clientpositive/parquet_create_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_create_gen_schema1.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/28372/diff/
>
>
> Testing
> -------
>
> Tested by adding appropriate qTests.
>
>
> Thanks,
>
> Ashish Singh
>
>
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde
to create table schema from a parquet file
Posted by Ashish Singh <as...@cloudera.com>.
> On Dec. 1, 2014, 8:13 p.m., Ryan Blue wrote:
> >
Ryan, thanks for the review. Made appropriate changes based on your comments. Kindly take a look.
- Ashish
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/#review63424
-----------------------------------------------------------
On Dec. 7, 2014, 1:40 a.m., Ashish Singh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28372/
> -----------------------------------------------------------
>
> (Updated Dec. 7, 2014, 1:40 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-8950
> https://issues.apache.org/jira/browse/HIVE-8950
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 2db2658fbc57fba01c892c9213baef6c498e659b
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java cd3d349e8bd8785d6cadaf9ed8fa7598f223774a
> ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/28372/diff/
>
>
> Testing
> -------
>
> Tested by adding appropriate qTests.
>
>
> Thanks,
>
> Ashish Singh
>
>
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde
to create table schema from a parquet file
Posted by Ashish Singh <as...@cloudera.com>.
> On Dec. 1, 2014, 8:13 p.m., Ryan Blue wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, line 762
> > <https://reviews.apache.org/r/28372/diff/2/?file=777120#file777120line762>
> >
> > What does this change do?
Serdes in this list must have one or more columns in the create table statement. This change removes this check for ParquetHiveSerDe.
> On Dec. 1, 2014, 8:13 p.m., Ryan Blue wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ColInfoFromParquetFile.java, line 36
> > <https://reviews.apache.org/r/28372/diff/2/?file=777121#file777121line36>
> >
> > I don't think ColInfoFromParquetFile is a great name for this class. What you've implemented here is a schema converter, like HiveSchemaConverter but from MessageType to TypeInfo rather than the other way around. I think this should either go in HiveSchemaConverter or a new class, like ParquetToHiveSchemaConverter.
> >
> > This update would also help clarify the methods exposed. I think that the primary method this should expose is:
> > StructType convert(GroupType parquetSchema)
> >
> > File reading should be done elsewhere if it is necessary.
> >
> > Also, I think this should return a StructType instead of a Pair with Strings. That would be a lot cleaner and would avoid the custom type string building immediately followed by parsing those type strings.
Made appropriate changes.
> On Dec. 1, 2014, 8:13 p.m., Ryan Blue wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java, line 107
> > <https://reviews.apache.org/r/28372/diff/2/?file=777122#file777122line107>
> >
> > Is it possible to avoid setting a file property? There are lots of cases where a file in the dataset would be removed so this is a brittle method of configuring the table. Ideally, we would check for the LIST_COLUMN_TYPES property and if we don't find it, conver the schema of the first file that we find.
Below are the different scenarios possible and how they will be handled.
1. Managed table with parquet.file specified:
1.a. While creating the table ParquetHiveSerDe will use that file to create hive schema.
1.b. If at a later point of time, say while select data from table, file specified by parquet.file is deleted, then ParquetHiveSerDe will look for a file in table dir and will try to create hive schema from that file's parquet metadata.
1.c. If parquet.file is deleted and table dir is empty, there is no way hive schema can be created. So, an informative exception will be thrown, "Either provide schema for table or point to parquet file using parquet.file in tblproperties or make sure that table has atleast one parquet file with required metadata". Note that if a table reaches this state, it can be fixed either by altering table and pointing parquet.file to a valid file or by copying a parquet file to table dir. This is valid for all cases.
2. Managed table without parquet.file specified:
2.a. While creating the table ParquetHiveSerDe will use any file in table dir to create hive schema. If table dir is empty, it will throw an informative exception.
3. External table with parquet.file specified: Same as (1).
4. External table without parquet.file specified and empty table dir: Same as (1.c)
5. External table without parquet.file specified and a parquet file present in table dir: Same as (2)
- Ashish
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/#review63424
-----------------------------------------------------------
On Nov. 27, 2014, 1:08 a.m., Ashish Singh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28372/
> -----------------------------------------------------------
>
> (Updated Nov. 27, 2014, 1:08 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-8950
> https://issues.apache.org/jira/browse/HIVE-8950
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 2db2658fbc57fba01c892c9213baef6c498e659b
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java cd3d349e8bd8785d6cadaf9ed8fa7598f223774a
> ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/28372/diff/
>
>
> Testing
> -------
>
> Tested by adding appropriate qTests.
>
>
> Thanks,
>
> Ashish Singh
>
>
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde
to create table schema from a parquet file
Posted by Ryan Blue <bl...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/#review63424
-----------------------------------------------------------
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
<https://reviews.apache.org/r/28372/#comment105680>
What does this change do?
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ColInfoFromParquetFile.java
<https://reviews.apache.org/r/28372/#comment105672>
I don't think ColInfoFromParquetFile is a great name for this class. What you've implemented here is a schema converter, like HiveSchemaConverter but from MessageType to TypeInfo rather than the other way around. I think this should either go in HiveSchemaConverter or a new class, like ParquetToHiveSchemaConverter.
This update would also help clarify the methods exposed. I think that the primary method this should expose is:
StructType convert(GroupType parquetSchema)
File reading should be done elsewhere if it is necessary.
Also, I think this should return a StructType instead of a Pair with Strings. That would be a lot cleaner and would avoid the custom type string building immediately followed by parsing those type strings.
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java
<https://reviews.apache.org/r/28372/#comment105675>
Could you please undo the import moves? I like to avoid non-functional changes.
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java
<https://reviews.apache.org/r/28372/#comment105676>
Is it possible to avoid setting a file property? There are lots of cases where a file in the dataset would be removed so this is a brittle method of configuring the table. Ideally, we would check for the LIST_COLUMN_TYPES property and if we don't find it, conver the schema of the first file that we find.
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java
<https://reviews.apache.org/r/28372/#comment105678>
This non-functional change is fine with me because it fixes the style in a method you're editing. But if you add a newline after this conditional, then you should also add one after line 149.
- Ryan Blue
On Nov. 26, 2014, 5:08 p.m., Ashish Singh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28372/
> -----------------------------------------------------------
>
> (Updated Nov. 26, 2014, 5:08 p.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-8950
> https://issues.apache.org/jira/browse/HIVE-8950
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ColInfoFromParquetFile.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd
> ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/28372/diff/
>
>
> Testing
> -------
>
> Tested by adding appropriate qTests.
>
>
> Thanks,
>
> Ashish Singh
>
>
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde
to create table schema from a parquet file
Posted by Ashish Singh <as...@cloudera.com>.
> On Dec. 11, 2014, 2:41 a.m., Brock Noland wrote:
> > This looks great!! I have a few comments below..
Thanks for the review. Addressed your comments.
> On Dec. 11, 2014, 2:41 a.m., Brock Noland wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java, line 96
> > <https://reviews.apache.org/r/28372/diff/7/?file=788512#file788512line96>
> >
> > Isn't this a timestamp?
It is :).
> On Dec. 11, 2014, 2:41 a.m., Brock Noland wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java, line 106
> > <https://reviews.apache.org/r/28372/diff/7/?file=788513#file788513line106>
> >
> > constants should be final
I guess you meant static
- Ashish
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/#review64671
-----------------------------------------------------------
On Dec. 13, 2014, 12:03 a.m., Ashish Singh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28372/
> -----------------------------------------------------------
>
> (Updated Dec. 13, 2014, 12:03 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-8950
> https://issues.apache.org/jira/browse/HIVE-8950
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 32186391e7e4cfc9b4d06d7376663e82ec08d9e6
> metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java b137fcb86e27ac91ed3c733b4d8788228d379a09
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 2db2658fbc57fba01c892c9213baef6c498e659b
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java cd3d349e8bd8785d6cadaf9ed8fa7598f223774a
> ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/results/clientpositive/create_view_partitioned.q.out ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a
> ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/28372/diff/
>
>
> Testing
> -------
>
> Tested by adding appropriate qTests.
>
>
> Thanks,
>
> Ashish Singh
>
>
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde
to create table schema from a parquet file
Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/#review64671
-----------------------------------------------------------
This looks great!! I have a few comments below..
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java
<https://reviews.apache.org/r/28372/#comment107467>
We should add a message such as
"Error reading footer from: " + parquetFilePath
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java
<https://reviews.apache.org/r/28372/#comment107473>
Isn't this a timestamp?
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java
<https://reviews.apache.org/r/28372/#comment107476>
pls comment on why we are not hanlding MAP_KEY_VALUE
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java
<https://reviews.apache.org/r/28372/#comment107475>
Why aren't we handling UTF8?
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java
<https://reviews.apache.org/r/28372/#comment107472>
constants should be final
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java
<https://reviews.apache.org/r/28372/#comment107471>
LOG shoudl be at the top of this clazz
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java
<https://reviews.apache.org/r/28372/#comment107468>
this should be error
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java
<https://reviews.apache.org/r/28372/#comment107469>
We should not printStackTrace. Please make the log statement:
LOG.error("Unable to read file from " + loc + ": " + e, e);
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
<https://reviews.apache.org/r/28372/#comment107470>
I don't think this is warnable. It's legal to specifify a location for an internal table.
- Brock Noland
On Dec. 10, 2014, 8:34 p.m., Ashish Singh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28372/
> -----------------------------------------------------------
>
> (Updated Dec. 10, 2014, 8:34 p.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-8950
> https://issues.apache.org/jira/browse/HIVE-8950
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 32186391e7e4cfc9b4d06d7376663e82ec08d9e6
> metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java b137fcb86e27ac91ed3c733b4d8788228d379a09
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 2db2658fbc57fba01c892c9213baef6c498e659b
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java cd3d349e8bd8785d6cadaf9ed8fa7598f223774a
> ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/results/clientpositive/create_view_partitioned.q.out ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a
> ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/28372/diff/
>
>
> Testing
> -------
>
> Tested by adding appropriate qTests.
>
>
> Thanks,
>
> Ashish Singh
>
>
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde
to create table schema from a parquet file
Posted by Ashish Singh <as...@cloudera.com>.
> On Dec. 16, 2014, 12:36 a.m., Brock Noland wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java, line 163
> > <https://reviews.apache.org/r/28372/diff/8/?file=790873#file790873line163>
> >
> > Now we are treating MAP_KEY_VALUE as a string. Is that correct??
My bad. Fixed!
- Ashish
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/#review65157
-----------------------------------------------------------
On Dec. 24, 2014, 4:19 a.m., Ashish Singh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28372/
> -----------------------------------------------------------
>
> (Updated Dec. 24, 2014, 4:19 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-8950
> https://issues.apache.org/jira/browse/HIVE-8950
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 32186391e7e4cfc9b4d06d7376663e82ec08d9e6
> metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java b137fcb86e27ac91ed3c733b4d8788228d379a09
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 2db2658fbc57fba01c892c9213baef6c498e659b
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java cd3d349e8bd8785d6cadaf9ed8fa7598f223774a
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetToHiveSchemaConverter.java PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/results/clientpositive/create_view_partitioned.q.out ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a
> ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/28372/diff/
>
>
> Testing
> -------
>
> Tested by adding appropriate qTests.
>
>
> Thanks,
>
> Ashish Singh
>
>
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde
to create table schema from a parquet file
Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/#review65157
-----------------------------------------------------------
Hi,
Thank you Ashish! I had just one concern, below.
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java
<https://reviews.apache.org/r/28372/#comment108147>
Now we are treating MAP_KEY_VALUE as a string. Is that correct??
- Brock Noland
On Dec. 13, 2014, 12:03 a.m., Ashish Singh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28372/
> -----------------------------------------------------------
>
> (Updated Dec. 13, 2014, 12:03 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-8950
> https://issues.apache.org/jira/browse/HIVE-8950
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 32186391e7e4cfc9b4d06d7376663e82ec08d9e6
> metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java b137fcb86e27ac91ed3c733b4d8788228d379a09
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 2db2658fbc57fba01c892c9213baef6c498e659b
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java cd3d349e8bd8785d6cadaf9ed8fa7598f223774a
> ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/results/clientpositive/create_view_partitioned.q.out ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a
> ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/28372/diff/
>
>
> Testing
> -------
>
> Tested by adding appropriate qTests.
>
>
> Thanks,
>
> Ashish Singh
>
>
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde
to create table schema from a parquet file
Posted by Ashish Singh <as...@cloudera.com>.
> On Dec. 13, 2014, 1:20 a.m., Ryan Blue wrote:
> > I don't know why I didn't catch this earlier, but I'd also like to see a unit test for the new schema converter.
Added unit tests.
> On Dec. 13, 2014, 1:20 a.m., Ryan Blue wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java, line 141
> > <https://reviews.apache.org/r/28372/diff/8/?file=790874#file790874line141>
> >
> > Why does this use a location property instead of the SD location?
tbl is just a Properties object, not actual Table object that has access to table's SD location.
> On Dec. 13, 2014, 1:20 a.m., Ryan Blue wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java, line 692
> > <https://reviews.apache.org/r/28372/diff/8/?file=790875#file790875line692>
> >
> > This appears to store the location in the storage descriptor, so I think that the javadoc is no longer correct.
Removed
> On Dec. 13, 2014, 1:20 a.m., Ryan Blue wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java, line 138
> > <https://reviews.apache.org/r/28372/diff/8/?file=790874#file790874line138>
> >
> > I'd rather not use parquet.file at all. Is there a compelling reason to keep it now that this can find files in the table location?
parquet.file or a similar table property will be required to create a parquet backed managed table. Hive expects atleast one column in the table, which can either come from user specified hive schema or from table's serde.
> On Dec. 13, 2014, 1:20 a.m., Ryan Blue wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 2294
> > <https://reviews.apache.org/r/28372/diff/8/?file=790870#file790870line2294>
> >
> > It isn't obvious why this is done or why deepCopy is used. Could you add a comment?
Done.
> On Dec. 13, 2014, 1:20 a.m., Ryan Blue wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java, line 96
> > <https://reviews.apache.org/r/28372/diff/8/?file=790873#file790873line96>
> >
> > I'm not sure we want to assume that an int96 is a timestamp. That's all it's used for today, but maybe we should make it an opt-in so that users must set the hive type to timestamp explicitly. That way, we keep from assuming that int96 will always be a timestamp or building behavior we might have to break later.
User can make hive assume parquet's int96 to map to hive's timestamp by setting "parquet.int96.is.timestamp" in table properties.
- Ashish
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/#review65004
-----------------------------------------------------------
On Dec. 13, 2014, 12:03 a.m., Ashish Singh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28372/
> -----------------------------------------------------------
>
> (Updated Dec. 13, 2014, 12:03 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-8950
> https://issues.apache.org/jira/browse/HIVE-8950
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 32186391e7e4cfc9b4d06d7376663e82ec08d9e6
> metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java b137fcb86e27ac91ed3c733b4d8788228d379a09
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 2db2658fbc57fba01c892c9213baef6c498e659b
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java cd3d349e8bd8785d6cadaf9ed8fa7598f223774a
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetToHiveSchemaConverter.java PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/results/clientpositive/create_view_partitioned.q.out ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a
> ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/28372/diff/
>
>
> Testing
> -------
>
> Tested by adding appropriate qTests.
>
>
> Thanks,
>
> Ashish Singh
>
>
Re: Review Request 28372: Address review comments
Posted by Ashish Singh <as...@cloudera.com>.
> On Dec. 13, 2014, 1:20 a.m., Ryan Blue wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java, line 138
> > <https://reviews.apache.org/r/28372/diff/8/?file=790874#file790874line138>
> >
> > I'd rather not use parquet.file at all. Is there a compelling reason to keep it now that this can find files in the table location?
>
> Ashish Singh wrote:
> parquet.file or a similar table property will be required to create a parquet backed managed table. Hive expects atleast one column in the table, which can either come from user specified hive schema or from table's serde.
Removed parquet.file and restricted the automatic schema generation to external tables poiniting to dir with parquet data.
- Ashish
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/#review65004
-----------------------------------------------------------
On April 17, 2015, 3:01 a.m., Ashish Singh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28372/
> -----------------------------------------------------------
>
> (Updated April 17, 2015, 3:01 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-8950
> https://issues.apache.org/jira/browse/HIVE-8950
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e138800e6dadd6fe76345f21eb76c906165c438d
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 53d913435cd43c96f044cf8668461fc686817ef4
> metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 3f267ff0eb20560c36a19b74353f9d6749c8b333
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 907ad2b8d7bae9b13eb4d9605ff2a3fe60c03ee8
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 7fd5e9612d4e3c9bf3b816bc48dbdbe59fb8a5a8
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java a029f10c116fd46070b2d41790043f0a7001390f
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetToHiveSchemaConverter.java PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/results/clientpositive/create_view_partitioned.q.out ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a
> ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/28372/diff/
>
>
> Testing
> -------
>
> Tested by adding appropriate qTests.
>
>
> Thanks,
>
> Ashish Singh
>
>
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde
to create table schema from a parquet file
Posted by Ryan Blue <bl...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/#review65004
-----------------------------------------------------------
I don't know why I didn't catch this earlier, but I'd also like to see a unit test for the new schema converter.
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
<https://reviews.apache.org/r/28372/#comment107915>
Should there be a comment here about why this was removed?
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
<https://reviews.apache.org/r/28372/#comment107917>
It isn't obvious why this is done or why deepCopy is used. Could you add a comment?
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java
<https://reviews.apache.org/r/28372/#comment107898>
This could easily be a Path object rather than a String. In fact, it would be more convenient for the caller also.
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java
<https://reviews.apache.org/r/28372/#comment107902>
This should throw IOException instead of coercing it to RuntimeException. This currently hides the fact that this might throw an Exception in the SerDe initialization, which is a bad thing. I think it makes sense to use the checked IOException and put a try/catch in initialize.
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java
<https://reviews.apache.org/r/28372/#comment107910>
I'm not sure we want to assume that an int96 is a timestamp. That's all it's used for today, but maybe we should make it an opt-in so that users must set the hive type to timestamp explicitly. That way, we keep from assuming that int96 will always be a timestamp or building behavior we might have to break later.
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java
<https://reviews.apache.org/r/28372/#comment107892>
I'd rather not use parquet.file at all. Is there a compelling reason to keep it now that this can find files in the table location?
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java
<https://reviews.apache.org/r/28372/#comment107920>
Why does this use a location property instead of the SD location?
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java
<https://reviews.apache.org/r/28372/#comment107911>
ParquetSchemaReader throws RuntimeException (should be IOException though) and the converter throws UnsupportedOperationException and IllegalArgumentException (should that be UnsupportedOperationException?).
I think this should have a try/catch with IOException and RuntimeException that prevents the initialization from failing.
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java
<https://reviews.apache.org/r/28372/#comment107912>
This is another exception that shouldn't be thrown.
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java
<https://reviews.apache.org/r/28372/#comment107897>
I think loc should be a Path rather than a String. This would avoid creating Path objects and converting them to Strings all the time.
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
<https://reviews.apache.org/r/28372/#comment107895>
This appears to store the location in the storage descriptor, so I think that the javadoc is no longer correct.
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
<https://reviews.apache.org/r/28372/#comment107893>
This will always be true. If the location is null, then the first branch of the containing conditional is always taken.
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
<https://reviews.apache.org/r/28372/#comment107894>
Why not return null here?
- Ryan Blue
On Dec. 12, 2014, 4:03 p.m., Ashish Singh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28372/
> -----------------------------------------------------------
>
> (Updated Dec. 12, 2014, 4:03 p.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-8950
> https://issues.apache.org/jira/browse/HIVE-8950
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 32186391e7e4cfc9b4d06d7376663e82ec08d9e6
> metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java b137fcb86e27ac91ed3c733b4d8788228d379a09
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 2db2658fbc57fba01c892c9213baef6c498e659b
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java cd3d349e8bd8785d6cadaf9ed8fa7598f223774a
> ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/results/clientpositive/create_view_partitioned.q.out ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a
> ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/28372/diff/
>
>
> Testing
> -------
>
> Tested by adding appropriate qTests.
>
>
> Thanks,
>
> Ashish Singh
>
>
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde
to create table schema from a parquet file
Posted by Ashish Singh <as...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/
-----------------------------------------------------------
(Updated April 30, 2015, 12:13 a.m.)
Review request for hive.
Changes
-------
Addressed Ryan's review comments.
Summary (updated)
-----------------
HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
Bugs: HIVE-8950
https://issues.apache.org/jira/browse/HIVE-8950
Repository: hive-git
Description
-------
HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 65ec1b94f081379d8c6abf0503b6c50305a35ef4
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 50e022da57575cc3020de34a0d34510f224a37de
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 3f267ff0eb20560c36a19b74353f9d6749c8b333
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java b89b07a7460e456918f7cb8054f8f2ff1a752033
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 7fd5e9612d4e3c9bf3b816bc48dbdbe59fb8a5a8
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 39c48b52665e4de024cf2abc99e0637549867fa3
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetToHiveSchemaConverter.java PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/results/clientpositive/create_view_partitioned.q.out ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a
ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/28372/diff/
Testing
-------
Tested by adding appropriate qTests.
Thanks,
Ashish Singh
Re: Review Request 28372: Address review comments
Posted by Ashish Singh <as...@cloudera.com>.
> On April 28, 2015, 9:30 p.m., Ryan Blue wrote:
> > Looking good to me. I found a few things to fix, and I'm also wondering where the qtest outputs are.
Thanks for the review Ryan!
qtest outputs are in *.q.out files.
- Ashish
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/#review81884
-----------------------------------------------------------
On April 17, 2015, 3:01 a.m., Ashish Singh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28372/
> -----------------------------------------------------------
>
> (Updated April 17, 2015, 3:01 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-8950
> https://issues.apache.org/jira/browse/HIVE-8950
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e138800e6dadd6fe76345f21eb76c906165c438d
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 53d913435cd43c96f044cf8668461fc686817ef4
> metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 3f267ff0eb20560c36a19b74353f9d6749c8b333
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 907ad2b8d7bae9b13eb4d9605ff2a3fe60c03ee8
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 7fd5e9612d4e3c9bf3b816bc48dbdbe59fb8a5a8
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java a029f10c116fd46070b2d41790043f0a7001390f
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetToHiveSchemaConverter.java PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/results/clientpositive/create_view_partitioned.q.out ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a
> ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/28372/diff/
>
>
> Testing
> -------
>
> Tested by adding appropriate qTests.
>
>
> Thanks,
>
> Ashish Singh
>
>
Re: Review Request 28372: Address review comments
Posted by Ryan Blue <bl...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/#review81884
-----------------------------------------------------------
Looking good to me. I found a few things to fix, and I'm also wondering where the qtest outputs are.
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java
<https://reviews.apache.org/r/28372/#comment132397>
Sergio discovered that MAP_KEY_VALUE was incorrectly used in place of MAP. So instead of throwing UnsupportedOperationException, MAP_KEY_VALUE should be used as a synonym for MAP. That way we can read old data.
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java
<https://reviews.apache.org/r/28372/#comment132398>
Shouldn't you also pass the exception so that the stack trace is printed?
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetToHiveSchemaConverter.java
<https://reviews.apache.org/r/28372/#comment132399>
Minor: Would be good to use the Types API instead of constructors.
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetToHiveSchemaConverter.java
<https://reviews.apache.org/r/28372/#comment132401>
I think the maps and lists should be constructed using the current best practice, ConversionPatterns, instead of by hand. That way we're testing what is actually going to be in data files.
- Ryan Blue
On April 16, 2015, 8:01 p.m., Ashish Singh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28372/
> -----------------------------------------------------------
>
> (Updated April 16, 2015, 8:01 p.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-8950
> https://issues.apache.org/jira/browse/HIVE-8950
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e138800e6dadd6fe76345f21eb76c906165c438d
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 53d913435cd43c96f044cf8668461fc686817ef4
> metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 3f267ff0eb20560c36a19b74353f9d6749c8b333
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 907ad2b8d7bae9b13eb4d9605ff2a3fe60c03ee8
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 7fd5e9612d4e3c9bf3b816bc48dbdbe59fb8a5a8
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java a029f10c116fd46070b2d41790043f0a7001390f
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetToHiveSchemaConverter.java PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/results/clientpositive/create_view_partitioned.q.out ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a
> ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/28372/diff/
>
>
> Testing
> -------
>
> Tested by adding appropriate qTests.
>
>
> Thanks,
>
> Ashish Singh
>
>
Re: Review Request 28372: Address review comments
Posted by Ashish Singh <as...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/
-----------------------------------------------------------
(Updated April 17, 2015, 3:01 a.m.)
Review request for hive.
Changes
-------
Removed parquet.file table property, restricting the capability of automatic schema generation to external tables.
Summary (updated)
-----------------
Address review comments
Bugs: HIVE-8950
https://issues.apache.org/jira/browse/HIVE-8950
Repository: hive-git
Description
-------
HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e138800e6dadd6fe76345f21eb76c906165c438d
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 53d913435cd43c96f044cf8668461fc686817ef4
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 3f267ff0eb20560c36a19b74353f9d6749c8b333
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 907ad2b8d7bae9b13eb4d9605ff2a3fe60c03ee8
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 7fd5e9612d4e3c9bf3b816bc48dbdbe59fb8a5a8
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java a029f10c116fd46070b2d41790043f0a7001390f
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetToHiveSchemaConverter.java PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/results/clientpositive/create_view_partitioned.q.out ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a
ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/28372/diff/
Testing
-------
Tested by adding appropriate qTests.
Thanks,
Ashish Singh
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde
to create table schema from a parquet file
Posted by Ashish Singh <as...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/
-----------------------------------------------------------
(Updated Dec. 24, 2014, 4:19 a.m.)
Review request for hive.
Changes
-------
Add unit tests
Bugs: HIVE-8950
https://issues.apache.org/jira/browse/HIVE-8950
Repository: hive-git
Description
-------
HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 32186391e7e4cfc9b4d06d7376663e82ec08d9e6
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java b137fcb86e27ac91ed3c733b4d8788228d379a09
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 2db2658fbc57fba01c892c9213baef6c498e659b
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java cd3d349e8bd8785d6cadaf9ed8fa7598f223774a
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetToHiveSchemaConverter.java PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/results/clientpositive/create_view_partitioned.q.out ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a
ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/28372/diff/
Testing
-------
Tested by adding appropriate qTests.
Thanks,
Ashish Singh
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde
to create table schema from a parquet file
Posted by Ashish Singh <as...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/
-----------------------------------------------------------
(Updated Dec. 13, 2014, 12:03 a.m.)
Review request for hive.
Changes
-------
Addressed review comments.
Bugs: HIVE-8950
https://issues.apache.org/jira/browse/HIVE-8950
Repository: hive-git
Description
-------
HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 32186391e7e4cfc9b4d06d7376663e82ec08d9e6
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java b137fcb86e27ac91ed3c733b4d8788228d379a09
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 2db2658fbc57fba01c892c9213baef6c498e659b
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java cd3d349e8bd8785d6cadaf9ed8fa7598f223774a
ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/results/clientpositive/create_view_partitioned.q.out ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a
ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/28372/diff/
Testing
-------
Tested by adding appropriate qTests.
Thanks,
Ashish Singh
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde
to create table schema from a parquet file
Posted by Ashish Singh <as...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/
-----------------------------------------------------------
(Updated Dec. 10, 2014, 8:34 p.m.)
Review request for hive.
Changes
-------
Address review comments.
Bugs: HIVE-8950
https://issues.apache.org/jira/browse/HIVE-8950
Repository: hive-git
Description
-------
HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 32186391e7e4cfc9b4d06d7376663e82ec08d9e6
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java b137fcb86e27ac91ed3c733b4d8788228d379a09
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 2db2658fbc57fba01c892c9213baef6c498e659b
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java cd3d349e8bd8785d6cadaf9ed8fa7598f223774a
ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/results/clientpositive/create_view_partitioned.q.out ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a
ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/28372/diff/
Testing
-------
Tested by adding appropriate qTests.
Thanks,
Ashish Singh
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde
to create table schema from a parquet file
Posted by Ashish Singh <as...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/
-----------------------------------------------------------
(Updated Dec. 10, 2014, 12:47 a.m.)
Review request for hive.
Changes
-------
Move to earlier version to pass table location to serde.
Bugs: HIVE-8950
https://issues.apache.org/jira/browse/HIVE-8950
Repository: hive-git
Description
-------
HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 32186391e7e4cfc9b4d06d7376663e82ec08d9e6
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java b137fcb86e27ac91ed3c733b4d8788228d379a09
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 2db2658fbc57fba01c892c9213baef6c498e659b
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java cd3d349e8bd8785d6cadaf9ed8fa7598f223774a
ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/results/clientpositive/create_view_partitioned.q.out ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a
ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/28372/diff/
Testing
-------
Tested by adding appropriate qTests.
Thanks,
Ashish Singh
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde
to create table schema from a parquet file
Posted by Ashish Singh <as...@cloudera.com>.
> On Dec. 9, 2014, 11:47 p.m., Ryan Blue wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java, line 144
> > <https://reviews.apache.org/r/28372/diff/5/?file=785269#file785269line144>
> >
> > There's a comment in the Avro SerDe that it can't throw an Exception because the SerDe is loaded when trying to change table properties. So an invalid table with a SerDe that throws an Exception can't be changed through the API. This is why Avro returns an invalid schema that is named something like "error_error_error_error_error_error_error". Here, maybe we can just return an empty schema?
It was my bad. Planned on doing it, but slipped out of my mind. Thanks for pointing this out.
- Ashish
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/#review64469
-----------------------------------------------------------
On Dec. 10, 2014, 8:34 p.m., Ashish Singh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28372/
> -----------------------------------------------------------
>
> (Updated Dec. 10, 2014, 8:34 p.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-8950
> https://issues.apache.org/jira/browse/HIVE-8950
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 32186391e7e4cfc9b4d06d7376663e82ec08d9e6
> metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java b137fcb86e27ac91ed3c733b4d8788228d379a09
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 2db2658fbc57fba01c892c9213baef6c498e659b
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java cd3d349e8bd8785d6cadaf9ed8fa7598f223774a
> ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/results/clientpositive/create_view_partitioned.q.out ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a
> ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/28372/diff/
>
>
> Testing
> -------
>
> Tested by adding appropriate qTests.
>
>
> Thanks,
>
> Ashish Singh
>
>
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde
to create table schema from a parquet file
Posted by Ryan Blue <bl...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/#review64469
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java
<https://reviews.apache.org/r/28372/#comment107187>
There's a comment in the Avro SerDe that it can't throw an Exception because the SerDe is loaded when trying to change table properties. So an invalid table with a SerDe that throws an Exception can't be changed through the API. This is why Avro returns an invalid schema that is named something like "error_error_error_error_error_error_error". Here, maybe we can just return an empty schema?
- Ryan Blue
On Dec. 7, 2014, 9:50 p.m., Ashish Singh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28372/
> -----------------------------------------------------------
>
> (Updated Dec. 7, 2014, 9:50 p.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-8950
> https://issues.apache.org/jira/browse/HIVE-8950
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 2db2658fbc57fba01c892c9213baef6c498e659b
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java cd3d349e8bd8785d6cadaf9ed8fa7598f223774a
> ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/28372/diff/
>
>
> Testing
> -------
>
> Tested by adding appropriate qTests.
>
>
> Thanks,
>
> Ashish Singh
>
>
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde
to create table schema from a parquet file
Posted by Ashish Singh <as...@cloudera.com>.
> On Dec. 9, 2014, 11:45 p.m., Ryan Blue wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java, line 111
> > <https://reviews.apache.org/r/28372/diff/5/?file=785268#file785268line111>
> >
> > I think the check for DECIMAL needs to be outside of this check. DECIMAL can be used to annotate int32, int64, fixed, and bytes. Catching it before this means we don't have to have this check in each convertX method.
> >
> > Also note that we validate the metadata as we read it from the file, so we can rely on the fact that if a field is annotated with DECIMAL, its type is compatible with DECIMAL.
Good point!
- Ashish
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/#review64467
-----------------------------------------------------------
On Dec. 10, 2014, 8:34 p.m., Ashish Singh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28372/
> -----------------------------------------------------------
>
> (Updated Dec. 10, 2014, 8:34 p.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-8950
> https://issues.apache.org/jira/browse/HIVE-8950
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 32186391e7e4cfc9b4d06d7376663e82ec08d9e6
> metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java b137fcb86e27ac91ed3c733b4d8788228d379a09
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 2db2658fbc57fba01c892c9213baef6c498e659b
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java cd3d349e8bd8785d6cadaf9ed8fa7598f223774a
> ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/results/clientpositive/create_view_partitioned.q.out ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a
> ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/28372/diff/
>
>
> Testing
> -------
>
> Tested by adding appropriate qTests.
>
>
> Thanks,
>
> Ashish Singh
>
>
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde
to create table schema from a parquet file
Posted by Ryan Blue <bl...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/#review64467
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java
<https://reviews.apache.org/r/28372/#comment107184>
I think the check for DECIMAL needs to be outside of this check. DECIMAL can be used to annotate int32, int64, fixed, and bytes. Catching it before this means we don't have to have this check in each convertX method.
Also note that we validate the metadata as we read it from the file, so we can rely on the fact that if a field is annotated with DECIMAL, its type is compatible with DECIMAL.
- Ryan Blue
On Dec. 7, 2014, 9:50 p.m., Ashish Singh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28372/
> -----------------------------------------------------------
>
> (Updated Dec. 7, 2014, 9:50 p.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-8950
> https://issues.apache.org/jira/browse/HIVE-8950
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 2db2658fbc57fba01c892c9213baef6c498e659b
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java cd3d349e8bd8785d6cadaf9ed8fa7598f223774a
> ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION
> ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/28372/diff/
>
>
> Testing
> -------
>
> Tested by adding appropriate qTests.
>
>
> Thanks,
>
> Ashish Singh
>
>
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde
to create table schema from a parquet file
Posted by Ashish Singh <as...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/
-----------------------------------------------------------
(Updated Dec. 8, 2014, 5:50 a.m.)
Review request for hive.
Changes
-------
Avoid changing hundreds of qtests' expected output by saving table location in table properties rather than serdeinfo.
Bugs: HIVE-8950
https://issues.apache.org/jira/browse/HIVE-8950
Repository: hive-git
Description
-------
HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 2db2658fbc57fba01c892c9213baef6c498e659b
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java cd3d349e8bd8785d6cadaf9ed8fa7598f223774a
ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/28372/diff/
Testing
-------
Tested by adding appropriate qTests.
Thanks,
Ashish Singh
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde
to create table schema from a parquet file
Posted by Ashish Singh <as...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/
-----------------------------------------------------------
(Updated Dec. 8, 2014, 1:06 a.m.)
Review request for hive.
Changes
-------
Fix test failures.
Bugs: HIVE-8950
https://issues.apache.org/jira/browse/HIVE-8950
Repository: hive-git
Description
-------
HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 2db2658fbc57fba01c892c9213baef6c498e659b
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java cd3d349e8bd8785d6cadaf9ed8fa7598f223774a
ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/results/clientnegative/alter_view_as_select_with_partition.q.out 447dc3aeba921a3c56ba587a52cd5624599b4499
ql/src/test/results/clientpositive/create_view_partitioned.q.out ebf9a6bc4f2321d7f539b7a445b3f279e3285b8a
ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/28372/diff/
Testing
-------
Tested by adding appropriate qTests.
Thanks,
Ashish Singh
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde
to create table schema from a parquet file
Posted by Ashish Singh <as...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/
-----------------------------------------------------------
(Updated Dec. 7, 2014, 1:40 a.m.)
Review request for hive.
Changes
-------
Addressed review comments.
Bugs: HIVE-8950
https://issues.apache.org/jira/browse/HIVE-8950
Repository: hive-git
Description
-------
HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 2db2658fbc57fba01c892c9213baef6c498e659b
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetSchemaReader.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ParquetToHiveSchemaConverter.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java cd3d349e8bd8785d6cadaf9ed8fa7598f223774a
ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema_ext.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema_ext.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/28372/diff/
Testing
-------
Tested by adding appropriate qTests.
Thanks,
Ashish Singh
Re: Review Request 28372: HIVE-8950: Add support in ParquetHiveSerde
to create table schema from a parquet file
Posted by Ashish Singh <as...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28372/
-----------------------------------------------------------
(Updated Nov. 27, 2014, 1:08 a.m.)
Review request for hive.
Changes
-------
Add more qtests.
Bugs: HIVE-8950
https://issues.apache.org/jira/browse/HIVE-8950
Repository: hive-git
Description
-------
HIVE-8950: Add support in ParquetHiveSerde to create table schema from a parquet file
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fafd78e63e9b41c9fdb0e017b567dc719d151784
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ColInfoFromParquetFile.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 4effe736fcf9d3715f03eed9885c299a7aa040dd
ql/src/test/queries/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_optional_elements_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_required_elements_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_structs_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_avro_array_of_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_decimal_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_multi_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_optional_elements_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_required_elements_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_structs_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_unannotated_groups_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_array_of_unannotated_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_avro_array_of_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_avro_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_decimal_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_thrift_array_of_primitives_gen_schema.q.out PRE-CREATION
ql/src/test/results/clientpositive/parquet_thrift_array_of_single_field_struct_gen_schema.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/28372/diff/
Testing
-------
Tested by adding appropriate qTests.
Thanks,
Ashish Singh