[jira] [Created] (PARQUET-2051) AvroWriteSupport does not pass Configuration to AvroSchemaConverter on Creation

["Andreas Hailu (Jira)" <ji...@apache.org>]

Andreas Hailu created PARQUET-2051:
--------------------------------------

             Summary: AvroWriteSupport does not pass Configuration to AvroSchemaConverter on Creation
                 Key: PARQUET-2051
                 URL: https://issues.apache.org/jira/browse/PARQUET-2051
             Project: Parquet
          Issue Type: Bug
            Reporter: Andreas Hailu


Because of this, we're unable to fully leverage the ThreeLevelListWriter functionality when trying to write Avro lists out using Parquet.

During testing, we see the following exception:
{quote}{{{{{{Caused by: java.lang.ClassCastException: repeated binary array (STRING) is not a group
        at org.apache.parquet.schema.Type.asGroupType(Type.java:250)
        at org.apache.parquet.avro.AvroWriteSupport$ThreeLevelListWriter.writeCollection(AvroWriteSupport.java:612)
        at org.apache.parquet.avro.AvroWriteSupport$ListWriter.writeList(AvroWriteSupport.java:397)
        at org.apache.parquet.avro.AvroWriteSupport.writeValueWithoutConversion(AvroWriteSupport.java:355)
        at org.apache.parquet.avro.AvroWriteSupport.writeValue(AvroWriteSupport.java:278)
        at org.apache.parquet.avro.AvroWriteSupport.writeRecordFields(AvroWriteSupport.java:191)
        at org.apache.parquet.avro.AvroWriteSupport.write(AvroWriteSupport.java:165)
        at org.apache.parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:128)}}}}}}
{quote}
Upon review, it was found that the configuration option that was set in `AvroWriteSupport` for the ThreeLevelListWriter, `parquet.avro.write-old-list-structure` being set to false, was never shared with the `AvroSchemaConverter`.

Once we made this change and tested locally, we observe the record with nulls in the array being successfully written by `AvroParquetOutputFormat`. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)