[jira] [Created] (FLINK-35620) Parquet writer creates wrong file for nested fields

["Vicky Papavasileiou (Jira)" <ji...@apache.org>]

Vicky Papavasileiou created FLINK-35620:
-------------------------------------------

             Summary: Parquet writer creates wrong file for nested fields
                 Key: FLINK-35620
                 URL: https://issues.apache.org/jira/browse/FLINK-35620
             Project: Flink
          Issue Type: Bug
          Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
    Affects Versions: 1.19.0
            Reporter: Vicky Papavasileiou


After PR [https://github.com/apache/flink/pull/24795] got merged that added support for nested arrays, the parquet writer produces wrong parquet files that cannot be read. Note, the reader (both flink and iceberg) don't throw an exception but return `null` for the nested field. 

The error is in how the field `max_definition_level` is populated for nested fields. 

Consider Avro schema:

```

{
"namespace": "com.test",
"type": "record",
"name": "RecordData",
"fields": [
{
"name": "Field1",
"type": {
"type": "array",
"items": {
"type": "record",
"name": "NestedField2",
"fields": [
{ "name": "NestedField3", "type": "double" }

]
}
}
}
]
}

```

Consider the excerpt below of a parquet file produced by Flink for the above schema:

```

############ Column(SegmentStartTime) ############
name: NestedField3
path: Field1.list.element.NestedField3
max_definition_level: 1
max_repetition_level: 1
physical_type: DOUBLE
logical_type: None
converted_type (legacy): NONE
compression: SNAPPY (space_saved: 7%)

```

The max_definition_level should be 4 but is 1



--
This message was sent by Atlassian Jira
(v8.20.10#820010)