You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Cheng Lian (JIRA)" <ji...@apache.org> on 2015/08/20 16:51:48 UTC
[jira] [Updated] (SPARK-10136) Parquet support fail to decode Avro
arrays of primitive array (e.g. array>)
[ https://issues.apache.org/jira/browse/SPARK-10136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheng Lian updated SPARK-10136:
-------------------------------
Description:
The following Avro schema
{noformat}
record AvroNonNullableArrays {
array<array<int>> int_arrays_column;
}
{noformat}
is translated into the following Parquet schema by parquet-avro 1.7.0:
{noformat}
message root {
required group int_arrays_column (LIST) {
repeated group array (LIST) {
repeated int32 array;
}
}
}
{noformat}
When making converters, the inner most {{array}} field is covered by {{RepeatedPrimitiveConverter}}, which is introduced in SPARK-9340 and [PR #8070|https://github.com/apache/spark/pull/8070]. However, PR #8070 assumes that repeated converters can only be created for struct fields, and doesn't take array elements into account.
was:
The following Avro schema
{noformat}
record AvroNonNullableArrays {
array<array<int>> int_arrays_column;
}
{noformat}
is translated into the following Parquet schema by parquet-avro 1.7.0:
{noformat}
message root {
required group int_arrays_column (LIST) {
repeated group array (LIST) {
repeated int32 array;
}
}
}
{noformat}
When making converters, the inner most {{array}} field is covered by {{RepeatedPrimitiveConverter}}, which is introduced in SPARK-9340 and [PR #8070|https://github.com/apache/spark/pull/8070]. However, PR #8070 assumes that repeated converters can only be created for struct fields, and doesn't take array elements into account.
> Parquet support fail to decode Avro arrays of primitive array (e.g. array<array<int>>)
> --------------------------------------------------------------------------------------
>
> Key: SPARK-10136
> URL: https://issues.apache.org/jira/browse/SPARK-10136
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.5.0
> Reporter: Cheng Lian
> Assignee: Cheng Lian
>
> The following Avro schema
> {noformat}
> record AvroNonNullableArrays {
> array<array<int>> int_arrays_column;
> }
> {noformat}
> is translated into the following Parquet schema by parquet-avro 1.7.0:
> {noformat}
> message root {
> required group int_arrays_column (LIST) {
> repeated group array (LIST) {
> repeated int32 array;
> }
> }
> }
> {noformat}
> When making converters, the inner most {{array}} field is covered by {{RepeatedPrimitiveConverter}}, which is introduced in SPARK-9340 and [PR #8070|https://github.com/apache/spark/pull/8070]. However, PR #8070 assumes that repeated converters can only be created for struct fields, and doesn't take array elements into account.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org