[jira] [Created] (PARQUET-465) Parquet-Avro does not support field removal

["Thomas Omans (JIRA)" <ji...@apache.org>]

Thomas Omans created PARQUET-465:
------------------------------------

             Summary: Parquet-Avro does not support field removal
                 Key: PARQUET-465
                 URL: https://issues.apache.org/jira/browse/PARQUET-465
             Project: Parquet
          Issue Type: Bug
          Components: parquet-avro
    Affects Versions: 1.8.0
            Reporter: Thomas Omans


Parquet avro does not support removal of fields, when used with the new compatibility layer:

Given a parquet file written with parquet avro at v1 and the following schema:

{code}
record FooBar {
  long foo;
  string bar;
}
{code}

And the following configuration settings:

{code}
job.getConfiguration.setBoolean(AvroReadSupport.AVRO_COMPATIBILITY, false)
AvroParquetInputFormat.setAvroReadSchema(job, avroReaderSchema)
{code}

A job fails when trying to read it using schema version v2:

{code}
record FooBar {
  string bar;
}
{code}

With the error:

{code}
org.apache.parquet.io.InvalidRecordException: Parquet/Avro schema mismatch: Avro field 'foo' not found
	at org.apache.parquet.avro.AvroRecordConverter.getAvroField(AvroRecordConverter.java:159)
{code}

It looks like because it sees the field in the original version it assumes the new version must expect it, but this case just means that the field was removed. Avro schema resolution dictates that you just ignore this field, since it is not relevant in the new version.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)