[jira] [Commented] (FLINK-32398) Support Avro SpecificRecord in DataStream and Table conversion.

["Jiangjie Qin (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-32398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735328#comment-17735328 ] 

Jiangjie Qin commented on FLINK-32398:
--------------------------------------

[~twalthr] [~jark] If this already works and I missed something, please let me know.

Orthogonal to this ticket, the {{AvroSchemaConverter }}class is not marked as Public, but from the commit log it seems this should be a public API. Same issue is there for the other classes in the flink-avro package. Actually non of the classes / interfaces is marked as Public in the entire flink-avro package.

> Support Avro SpecificRecord in DataStream and Table conversion.
> ---------------------------------------------------------------
>
>                 Key: FLINK-32398
>                 URL: https://issues.apache.org/jira/browse/FLINK-32398
>             Project: Flink
>          Issue Type: New Feature
>          Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>    Affects Versions: 1.17.1
>            Reporter: Jiangjie Qin
>            Priority: Major
>
> At this point, it seems that Avro SpecificRecord is not supported in DataStream and Table conversion. For example, the following code breaks when MyAvroRecord contains fields of type Record, Enum, Array, etc.
>  
> {code:java}
> ing schemaString = MyAvroRecord.getClassSchema().toString();
> DataType dataType = AvroSchemaConverter.convertToDataType(schemaString);
> TypeInformation<MyAvroRecord> typeInfo = AvroSchemaConverter.convertToTypeInfo(schemaString);;
> input.getTransformation().setOutputType(typeInfo);
> tEnv.createTemporaryView("myTable", input);
> Table result = tEnv.sqlQuery("SELECT * FROM myTable");
> DataStream<MyAvroRecord> output = tEnv.toDataStream(result, dataType);
> output.getTransformation().setOutputType(typeInfo); {code}
>  
> While the conversion from {{MyAvroRecord}} to {{RowData}} seems fine, several issues were there when converting the {{RowData}} back to {{{}MyAvroRecord{}}}, including but not limited to:
>  # {{AvroSchemaConverter.convertToDataType(schema)}} maps Avro Record type to RowType, which loses the class information.
>  # {{AvroSchemaConverter}} maps Enum to StringType, and simply try to cast the string to the Enum.
> I did not find a way to easily convert the between DataStream and Table for Avro SpecificRecord. Given the popularity of Avro SpecificRecord, we should support this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)