[jira] [Commented] (PARQUET-364) Parque-avro cannot decode Avro/Thrift array of primitive array (e.g. array>)

["Cheng Lian (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PARQUET-364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706959#comment-14706959 ] 

Cheng Lian commented on PARQUET-364:
------------------------------------

I tested the Thrift case with Thrift 0.9.2 because I can't get Thrift 0.7.0 compiled on Mac OS X 10.10 because of lacking proper C++ header files. I assume that this doesn't change the essence of this issue.

(BTW, any plan to upgrade to Thrift 0.9.2?)

> Parque-avro cannot decode Avro/Thrift array of primitive array (e.g. array<array<int>>)
> ---------------------------------------------------------------------------------------
>
>                 Key: PARQUET-364
>                 URL: https://issues.apache.org/jira/browse/PARQUET-364
>             Project: Parquet
>          Issue Type: Bug
>          Components: parquet-mr
>    Affects Versions: 1.5.0, 1.6.0, 1.7.0, 1.8.0
>            Reporter: Cheng Lian
>
> The problematic Avro and Thrift schemas are:
> {noformat}
> record AvroArrayOfArray {
>   array<array<int>> int_arrays_column;
> }
> {noformat}
> and
> {noformat}
> struct ThriftListOfList {
>   1: list<list<i32>> intArraysColumn;
> }
> {noformat}
> They are converted to the following Parquet schemas by parquet-avro 1.7.0 and parquet-thrift 1.7.0 respectively:
> {noformat}
> message AvroArrayOfArray {
>   required group int_arrays_column (LIST) {
>     repeated group array (LIST) {
>       repeated int32 array;
>     }
>   }
> }
> {noformat}
> and
> {noformat}
> message ParquetSchema {
>   required group intListsColumn (LIST) {
>     repeated group intListsColumn_tuple (LIST) {
>       repeated int32 intListsColumn_tuple_tuple;
>     }
>   }
> }
> {noformat}
> {{AvroIndexedRecordConverter}} cannot decode such records correctly. The reason is that the 2nd level repeated group {{array}} doesn't pass {{AvroIndexedRecordConverter.isElementType()}} check. We should check for field name "array" and field name suffix "_thrift" in {{isElementType()}} to fix this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)