[jira] [Created] (FLINK-40264) nested legacyTimestampMapping

["Pritam Kumar (Jira)" <ji...@apache.org>]

Pritam Kumar created FLINK-40264:
------------------------------------

             Summary: nested legacyTimestampMapping
                 Key: FLINK-40264
                 URL: https://issues.apache.org/jira/browse/FLINK-40264
             Project: Flink
          Issue Type: Bug
          Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
    Affects Versions: 2.3.0, 2.2.0, 1.20.0
            Reporter: Pritam Kumar


h2. Problem

  {{AvroToRowDataConverters#createConverter}} dispatches a nested {{ROW}} to the single argument overload:

  {code:java}
  case ROW:
      return createRowConverter((RowType) type);
  {code}

  {{createRowConverter(RowType)}} defaults {{legacyTimestampMapping}} to {{true}}, so the flag is dropped for every nesting level below the top one. Since {{TIMESTAMP_LTZ}} is only supported by the non legacy mapping, a nested {{TIMESTAMP_LTZ}}
  column fails even when the non legacy mapping was explicitly requested:

  {noformat}
  java.lang.UnsupportedOperationException: Unsupported type: TIMESTAMP_LTZ(3) NOT NULL
  {noformat}

  The failure happens while the deserialization schema is being constructed, so the job never starts.

  h2. Reproduction

  {code:java}
  DataType dataType =
          ROW(
                  FIELD("id", INT().notNull()),
                  FIELD("nested", ROW(FIELD("ltz3", TIMESTAMP_LTZ(3).notNull())).notNull()))
          .notNull();
  RowType rowType = (RowType) dataType.getLogicalType();

  new AvroRowDataDeserializationSchema(
          rowType,
          InternalTypeInfo.of(rowType),
          AvroEncoding.BINARY,
          false /* legacyTimestampMapping */);   // throws
  {code}

  {{AvroSchemaConverter#convertToSchema}} handles the same type correctly with {{legacyTimestampMapping = false}}, and {{RowDataToAvroConverters}} propagates the flag properly, so only the deserialization side is affected.

  h2. Fix

  Propagate the flag:

  {code:java}
  case ROW:
      return createRowConverter((RowType) type, legacyTimestampMapping);
  {code}

  Strictly a widening: the only code paths whose behaviour changes are ones that throw today.

  h2. Introduced by

  FLINK-33198, first released in 1.19.0.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)