You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Raymond Lau <ra...@gmail.com> on 2014/08/13 02:06:20 UTC
ArrayWritableGroupConverter
Hello. (First off, sorry if I accidentally posted to the wrong mailing
list before - dev - and you are getting this again)
Regarding the ArrayWritableGroupConverter class: I was just wondering how
come the field count has to be either 1 or 2? I'm trying to read a column
where the amount is fields is 3 and I'm getting an invalid parquet hive
schema (in hive 0.12) error when I try to do so. It looks like it links
back to here.
*https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ArrayWritableGroupConverter.java
<https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ArrayWritableGroupConverter.java>*
Thanks,
-Raymond
Re: ArrayWritableGroupConverter
Posted by Raymond Lau <rl...@ooyala.com>.
Anyone know anything about this? We have our errors stored in this data
structure, so it's hard to debug when we can't analyze our errors :\
Shameless bump.
On Thu, Aug 14, 2014 at 10:43 AM, Raymond Lau <rl...@ooyala.com> wrote:
> Original Thrift schema:
>
> struct teststruct {
> 1: optional string field1;
> 2: optional string field2;
> 3: optional string field3;
> }
>
> struct mainstruct {
> 1: optional list<teststruct> teststructs;
> }
>
> This parquet file schema was generated:
>
> message ParquetSchema {
> optional group teststructs {
> repeated group teststruct_tuple {
> optional binary field1;
> optional binary field2;
> optional binary field3;
> }
> }
> }
>
> When i try to run queries involving this 'teststructs' column, I get this
> error:
>
> Failed with exception java.io.IOException:java.lang.RuntimeException:
> Invalid parquet hive schema: repeated group teststruct_tuple {
> optional binary field1;
> optional binary field2;
> optional binary field3;
> }
>
>
>
> On Thu, Aug 14, 2014 at 8:35 AM, Brock Noland <br...@cloudera.com> wrote:
>
>> Hi,
>>
>> Can you share your parquet schema?
>>
>> Brock
>>
>>
>> On Tue, Aug 12, 2014 at 5:06 PM, Raymond Lau <ra...@gmail.com>
>> wrote:
>>
>>> Hello. (First off, sorry if I accidentally posted to the wrong mailing
>>> list before - dev - and you are getting this again)
>>>
>>> Regarding the ArrayWritableGroupConverter class: I was just wondering
>>> how come the field count has to be either 1 or 2? I'm trying to read a
>>> column where the amount is fields is 3 and I'm getting an invalid parquet
>>> hive schema (in hive 0.12) error when I try to do so. It looks like it
>>> links back to here.
>>>
>>> *https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ArrayWritableGroupConverter.java
>>> <https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ArrayWritableGroupConverter.java>*
>>>
>>>
>>> Thanks,
>>> -Raymond
>>>
>>
>>
>
>
> --
> *Raymond Lau*
> Software Engineer - Intern |
> rlau@ooyala.com | (925) 395-3806
>
--
*Raymond Lau*
Software Engineer - Intern |
rlau@ooyala.com | (925) 395-3806
Re: ArrayWritableGroupConverter
Posted by Raymond Lau <rl...@ooyala.com>.
Original Thrift schema:
struct teststruct {
1: optional string field1;
2: optional string field2;
3: optional string field3;
}
struct mainstruct {
1: optional list<teststruct> teststructs;
}
This parquet file schema was generated:
message ParquetSchema {
optional group teststructs {
repeated group teststruct_tuple {
optional binary field1;
optional binary field2;
optional binary field3;
}
}
}
When i try to run queries involving this 'teststructs' column, I get this
error:
Failed with exception java.io.IOException:java.lang.RuntimeException:
Invalid parquet hive schema: repeated group teststruct_tuple {
optional binary field1;
optional binary field2;
optional binary field3;
}
On Thu, Aug 14, 2014 at 8:35 AM, Brock Noland <br...@cloudera.com> wrote:
> Hi,
>
> Can you share your parquet schema?
>
> Brock
>
>
> On Tue, Aug 12, 2014 at 5:06 PM, Raymond Lau <ra...@gmail.com>
> wrote:
>
>> Hello. (First off, sorry if I accidentally posted to the wrong mailing
>> list before - dev - and you are getting this again)
>>
>> Regarding the ArrayWritableGroupConverter class: I was just wondering how
>> come the field count has to be either 1 or 2? I'm trying to read a column
>> where the amount is fields is 3 and I'm getting an invalid parquet hive
>> schema (in hive 0.12) error when I try to do so. It looks like it links
>> back to here.
>>
>> *https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ArrayWritableGroupConverter.java
>> <https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ArrayWritableGroupConverter.java>*
>>
>>
>> Thanks,
>> -Raymond
>>
>
>
--
*Raymond Lau*
Software Engineer - Intern |
rlau@ooyala.com | (925) 395-3806
Re: ArrayWritableGroupConverter
Posted by Brock Noland <br...@cloudera.com>.
Hi,
Can you share your parquet schema?
Brock
On Tue, Aug 12, 2014 at 5:06 PM, Raymond Lau <ra...@gmail.com>
wrote:
> Hello. (First off, sorry if I accidentally posted to the wrong mailing
> list before - dev - and you are getting this again)
>
> Regarding the ArrayWritableGroupConverter class: I was just wondering how
> come the field count has to be either 1 or 2? I'm trying to read a column
> where the amount is fields is 3 and I'm getting an invalid parquet hive
> schema (in hive 0.12) error when I try to do so. It looks like it links
> back to here.
>
> *https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ArrayWritableGroupConverter.java
> <https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ArrayWritableGroupConverter.java>*
>
>
> Thanks,
> -Raymond
>