You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Kevin Lam <ke...@shopify.com> on 2022/03/09 16:04:18 UTC

Evolving Schemas with ParquetColumnarRowInputFormat

Hi all,

We're interested in using ParquetColumnarRowInputFormat
<https://nightlies.apache.org/flink/flink-docs-release-1.14/api/java/org/apache/flink/formats/parquet/ParquetColumnarRowInputFormat.html>
or similar with evolving Parquet schemas. Any advice or recommendations?

Specifically, the situation we are interested in is when the passed in
RowType projectedType contains a new field with Type.Repetition.OPTIONAL
that is not present in the Parquet file being read. In this case we want
that column to just be read as null.

Thanks in advance for your help!