You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2015/08/21 19:52:45 UTC
[jira] [Commented] (SPARK-10136) Parquet support fail to decode
Avro/Thrift arrays of primitive array (e.g. array>)
[ https://issues.apache.org/jira/browse/SPARK-10136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14707137#comment-14707137 ]
Apache Spark commented on SPARK-10136:
--------------------------------------
User 'liancheng' has created a pull request for this issue:
https://github.com/apache/spark/pull/8361
> Parquet support fail to decode Avro/Thrift arrays of primitive array (e.g. array<array<int>>)
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-10136
> URL: https://issues.apache.org/jira/browse/SPARK-10136
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.5.0
> Reporter: Cheng Lian
> Assignee: Cheng Lian
> Priority: Blocker
> Fix For: 1.5.0
>
>
> The following Avro schema
> {noformat}
> record AvroNonNullableArrays {
> array<array<int>> int_arrays_column;
> }
> {noformat}
> is translated into the following Parquet schema by parquet-avro 1.7.0:
> {noformat}
> message root {
> required group int_arrays_column (LIST) {
> repeated group array (LIST) {
> repeated int32 array;
> }
> }
> }
> {noformat}
> When making converters, the inner most {{array}} field is covered by {{RepeatedPrimitiveConverter}}, which is introduced in SPARK-9340 and [PR #8070|https://github.com/apache/spark/pull/8070]. However, PR #8070 assumes that repeated converters can only be created for struct fields, and doesn't take array elements into account.
> Verified that 1.4 doesn't suffer this issue. This is a 1.5 regression.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org