[GitHub] [hudi] hanjun996 commented on issue #4698: [SUPPORT] RowDataToAvroConverters does not support data in flink timestamp_ltz (timestamp_with_local_time_zone) format.

[GitBox <gi...@apache.org>]

hanjun996 commented on issue #4698:
URL: https://github.com/apache/hudi/issues/4698#issuecomment-1031040459


   Environment Description
   Hudi version :0.10.0
   flink version :1.13.0
   
   
   
   
   Steps to reproduce the behavior:
   
   
   mysql timestamp type, such as: 1970-01-01T00:00:01Z, the corresponding type in flink is TIMESTAMP_WITH_LOCAL_TIME_ZONE, which is not supported by avro during conversion in hudi version 0.10.0
   
   code：
   
       public static final DataType ROW_DATA_TYPE = DataTypes.ROW(
           DataTypes.FIELD("id", DataTypes.STRING()),
           DataTypes.FIELD("timestamp", DataTypes.TIMESTAMP_WITH_LOCAL_TIME_ZONE()))
           .notNull();
       public static final RowType ROW_TYPE = (RowType) ROW_DATA_TYPE.getLogicalType();
   
       @Test
       public void test() {
           String inferredSchema = AvroSchemaConverter.convertToSchema(ROW_TYPE).toString();
           Configuration configuration = new Configuration();
           configuration.setString(FlinkOptions.PATH, "test");
           configuration.setString(FlinkOptions.TABLE_NAME, "test");
           configuration.setString(FlinkOptions.TABLE_TYPE, "MERGE_ON_READ");
           configuration.setString(FlinkOptions.RECORD_KEY_FIELD, "id");
           configuration.setString(FlinkOptions.SOURCE_AVRO_SCHEMA, inferredSchema);
   
           Map<String, String> record1 = new HashMap<>();
           record1.put("id", "1");
           record1.put("timestamp", "1970-01-01T00:00:01Z");
   
           StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
           DataStreamSource source = env.fromElements(record1);
           Pipelines.bootstrap(configuration, ROW_TYPE, 1, source).print();
       }
   	
   ERROR:
   
   java.lang.UnsupportedOperationException: Unsupported to derive Schema for type: TIMESTAMP_LTZ(6)
   	at org.apache.hudi.util.AvroSchemaConverter.convertToSchema(AvroSchemaConverter.java:292)
   	at org.apache.hudi.util.AvroSchemaConverter.convertToSchema(AvroSchemaConverter.java:263)
   	at org.apache.hudi.util.AvroSchemaConverter.convertToSchema(AvroSchemaConverter.java:169)
   
   
   
   
   Expected behavior:
   
   support TIMESTAMP_WITH_LOCAL_TIME_ZONE
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org