You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/01/27 06:59:12 UTC
[GitHub] [hudi] hanjun996 opened a new issue #4698: [SUPPORT] 83/5000 拼音 翻译 RowDataToAvroConverters does not support data in flink timestamp_ltz (timestamp_with_local_time_zone) format.
hanjun996 opened a new issue #4698:
URL: https://github.com/apache/hudi/issues/4698
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
A clear and concise description of the problem.
**To Reproduce**
Steps to reproduce the behavior:
1.
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version :
* Spark version :
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) :
* Running on Docker? (yes/no) :
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #4698: [SUPPORT] RowDataToAvroConverters does not support data in flink timestamp_ltz (timestamp_with_local_time_zone) format.
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #4698:
URL: https://github.com/apache/hudi/issues/4698#issuecomment-1032273542
Yeah, the timestamp_LTZ is not supported yet for flink, i have fired a JIRA to support this feature: https://issues.apache.org/jira/browse/HUDI-3388
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hanjun996 removed a comment on issue #4698: [SUPPORT] RowDataToAvroConverters does not support data in flink timestamp_ltz (timestamp_with_local_time_zone) format.
Posted by GitBox <gi...@apache.org>.
hanjun996 removed a comment on issue #4698:
URL: https://github.com/apache/hudi/issues/4698#issuecomment-1031040459
Environment Description
Hudi version :0.10.0
flink version :1.13.0
Steps to reproduce the behavior:
mysql timestamp type, such as: 1970-01-01T00:00:01Z, the corresponding type in flink is TIMESTAMP_WITH_LOCAL_TIME_ZONE, which is not supported by avro during conversion in hudi version 0.10.0
code：
public static final DataType ROW_DATA_TYPE = DataTypes.ROW(
DataTypes.FIELD("id", DataTypes.STRING()),
DataTypes.FIELD("timestamp", DataTypes.TIMESTAMP_WITH_LOCAL_TIME_ZONE()))
.notNull();
public static final RowType ROW_TYPE = (RowType) ROW_DATA_TYPE.getLogicalType();
@Test
public void test() {
String inferredSchema = AvroSchemaConverter.convertToSchema(ROW_TYPE).toString();
Configuration configuration = new Configuration();
configuration.setString(FlinkOptions.PATH, "test");
configuration.setString(FlinkOptions.TABLE_NAME, "test");
configuration.setString(FlinkOptions.TABLE_TYPE, "MERGE_ON_READ");
configuration.setString(FlinkOptions.RECORD_KEY_FIELD, "id");
configuration.setString(FlinkOptions.SOURCE_AVRO_SCHEMA, inferredSchema);
Map<String, String> record1 = new HashMap<>();
record1.put("id", "1");
record1.put("timestamp", "1970-01-01T00:00:01Z");
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
DataStreamSource source = env.fromElements(record1);
Pipelines.bootstrap(configuration, ROW_TYPE, 1, source).print();
}
ERROR:
java.lang.UnsupportedOperationException: Unsupported to derive Schema for type: TIMESTAMP_LTZ(6)
at org.apache.hudi.util.AvroSchemaConverter.convertToSchema(AvroSchemaConverter.java:292)
at org.apache.hudi.util.AvroSchemaConverter.convertToSchema(AvroSchemaConverter.java:263)
at org.apache.hudi.util.AvroSchemaConverter.convertToSchema(AvroSchemaConverter.java:169)
Expected behavior:
support TIMESTAMP_WITH_LOCAL_TIME_ZONE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #4698: [SUPPORT] RowDataToAvroConverters does not support data in flink timestamp_ltz (timestamp_with_local_time_zone) format.
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #4698:
URL: https://github.com/apache/hudi/issues/4698
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hanjun996 commented on issue #4698: [SUPPORT] RowDataToAvroConverters does not support data in flink timestamp_ltz (timestamp_with_local_time_zone) format.
Posted by GitBox <gi...@apache.org>.
hanjun996 commented on issue #4698:
URL: https://github.com/apache/hudi/issues/4698#issuecomment-1031041107
> ### **[hanjun996](/hanjun996) ** commented [33秒钟前](#issuecomment-1031040459)
> Environment Description
Hudi version :0.10.0
flink version :1.13.0
Steps to reproduce the behavior:
mysql timestamp type, such as: 1970-01-01T00:00:01Z, the corresponding type in flink is TIMESTAMP_WITH_LOCAL_TIME_ZONE, which is not supported by avro during conversion in hudi version 0.10.0
code：
public static final DataType ROW_DATA_TYPE = DataTypes.ROW(
DataTypes.FIELD("id", DataTypes.STRING()),
DataTypes.FIELD("timestamp", DataTypes.TIMESTAMP_WITH_LOCAL_TIME_ZONE()))
.notNull();
public static final RowType ROW_TYPE = (RowType) ROW_DATA_TYPE.getLogicalType();
@Test
public void test() {
String inferredSchema = AvroSchemaConverter.convertToSchema(ROW_TYPE).toString();
Configuration configuration = new Configuration();
configuration.setString(FlinkOptions.PATH, "test");
configuration.setString(FlinkOptions.TABLE_NAME, "test");
configuration.setString(FlinkOptions.TABLE_TYPE, "MERGE_ON_READ");
configuration.setString(FlinkOptions.RECORD_KEY_FIELD, "id");
configuration.setString(FlinkOptions.SOURCE_AVRO_SCHEMA, inferredSchema);
Map<String, String> record1 = new HashMap<>();
record1.put("id", "1");
record1.put("timestamp", "1970-01-01T00:00:01Z");
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
DataStreamSource source = env.fromElements(record1);
Pipelines.bootstrap(configuration, ROW_TYPE, 1, source).print();
}
ERROR:
java.lang.UnsupportedOperationException: Unsupported to derive Schema for type: TIMESTAMP_LTZ(6)
at org.apache.hudi.util.AvroSchemaConverter.convertToSchema(AvroSchemaConverter.java:292)
at org.apache.hudi.util.AvroSchemaConverter.convertToSchema(AvroSchemaConverter.java:263)
at org.apache.hudi.util.AvroSchemaConverter.convertToSchema(AvroSchemaConverter.java:169)
Expected behavior:
support TIMESTAMP_WITH_LOCAL_TIME_ZONE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #4698: [SUPPORT] RowDataToAvroConverters does not support data in flink timestamp_ltz (timestamp_with_local_time_zone) format.
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #4698:
URL: https://github.com/apache/hudi/issues/4698
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #4698: [SUPPORT] RowDataToAvroConverters does not support data in flink timestamp_ltz (timestamp_with_local_time_zone) format.
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #4698:
URL: https://github.com/apache/hudi/issues/4698#issuecomment-1025233718
@hanjun996 can you share environment configs, software versions, and code snippet to help us reproduce the issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hanjun996 commented on issue #4698: [SUPPORT] RowDataToAvroConverters does not support data in flink timestamp_ltz (timestamp_with_local_time_zone) format.
Posted by GitBox <gi...@apache.org>.
hanjun996 commented on issue #4698:
URL: https://github.com/apache/hudi/issues/4698#issuecomment-1031040459
Environment Description
Hudi version :0.10.0
flink version :1.13.0
Steps to reproduce the behavior:
mysql timestamp type, such as: 1970-01-01T00:00:01Z, the corresponding type in flink is TIMESTAMP_WITH_LOCAL_TIME_ZONE, which is not supported by avro during conversion in hudi version 0.10.0
code：
public static final DataType ROW_DATA_TYPE = DataTypes.ROW(
DataTypes.FIELD("id", DataTypes.STRING()),
DataTypes.FIELD("timestamp", DataTypes.TIMESTAMP_WITH_LOCAL_TIME_ZONE()))
.notNull();
public static final RowType ROW_TYPE = (RowType) ROW_DATA_TYPE.getLogicalType();
@Test
public void test() {
String inferredSchema = AvroSchemaConverter.convertToSchema(ROW_TYPE).toString();
Configuration configuration = new Configuration();
configuration.setString(FlinkOptions.PATH, "test");
configuration.setString(FlinkOptions.TABLE_NAME, "test");
configuration.setString(FlinkOptions.TABLE_TYPE, "MERGE_ON_READ");
configuration.setString(FlinkOptions.RECORD_KEY_FIELD, "id");
configuration.setString(FlinkOptions.SOURCE_AVRO_SCHEMA, inferredSchema);
Map<String, String> record1 = new HashMap<>();
record1.put("id", "1");
record1.put("timestamp", "1970-01-01T00:00:01Z");
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
DataStreamSource source = env.fromElements(record1);
Pipelines.bootstrap(configuration, ROW_TYPE, 1, source).print();
}
ERROR:
java.lang.UnsupportedOperationException: Unsupported to derive Schema for type: TIMESTAMP_LTZ(6)
at org.apache.hudi.util.AvroSchemaConverter.convertToSchema(AvroSchemaConverter.java:292)
at org.apache.hudi.util.AvroSchemaConverter.convertToSchema(AvroSchemaConverter.java:263)
at org.apache.hudi.util.AvroSchemaConverter.convertToSchema(AvroSchemaConverter.java:169)
Expected behavior:
support TIMESTAMP_WITH_LOCAL_TIME_ZONE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4698: [SUPPORT] RowDataToAvroConverters does not support data in flink timestamp_ltz (timestamp_with_local_time_zone) format.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4698:
URL: https://github.com/apache/hudi/issues/4698#issuecomment-1039622562
thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hanjun996 edited a comment on issue #4698: [SUPPORT] RowDataToAvroConverters does not support data in flink timestamp_ltz (timestamp_with_local_time_zone) format.
Posted by GitBox <gi...@apache.org>.
hanjun996 edited a comment on issue #4698:
URL: https://github.com/apache/hudi/issues/4698#issuecomment-1031041107
> Environment Description
Hudi version :0.10.0
flink version :1.13.0
Steps to reproduce the behavior:
mysql timestamp type, such as: 1970-01-01T00:00:01Z, the corresponding type in flink is TIMESTAMP_WITH_LOCAL_TIME_ZONE, which is not supported by avro during conversion in hudi version 0.10.0
code：
public static final DataType ROW_DATA_TYPE = DataTypes.ROW(
DataTypes.FIELD("id", DataTypes.STRING()),
DataTypes.FIELD("timestamp", DataTypes.TIMESTAMP_WITH_LOCAL_TIME_ZONE()))
.notNull();
public static final RowType ROW_TYPE = (RowType) ROW_DATA_TYPE.getLogicalType();
@Test
public void test() {
String inferredSchema = AvroSchemaConverter.convertToSchema(ROW_TYPE).toString();
Configuration configuration = new Configuration();
configuration.setString(FlinkOptions.PATH, "test");
configuration.setString(FlinkOptions.TABLE_NAME, "test");
configuration.setString(FlinkOptions.TABLE_TYPE, "MERGE_ON_READ");
configuration.setString(FlinkOptions.RECORD_KEY_FIELD, "id");
configuration.setString(FlinkOptions.SOURCE_AVRO_SCHEMA, inferredSchema);
Map<String, String> record1 = new HashMap<>();
record1.put("id", "1");
record1.put("timestamp", "1970-01-01T00:00:01Z");
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
DataStreamSource source = env.fromElements(record1);
Pipelines.bootstrap(configuration, ROW_TYPE, 1, source).print();
}
ERROR:
java.lang.UnsupportedOperationException: Unsupported to derive Schema for type: TIMESTAMP_LTZ(6)
at org.apache.hudi.util.AvroSchemaConverter.convertToSchema(AvroSchemaConverter.java:292)
at org.apache.hudi.util.AvroSchemaConverter.convertToSchema(AvroSchemaConverter.java:263)
at org.apache.hudi.util.AvroSchemaConverter.convertToSchema(AvroSchemaConverter.java:169)
Expected behavior:
support TIMESTAMP_WITH_LOCAL_TIME_ZONE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4698: [SUPPORT] RowDataToAvroConverters does not support data in flink timestamp_ltz (timestamp_with_local_time_zone) format.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4698:
URL: https://github.com/apache/hudi/issues/4698#issuecomment-1032003895
@danny0405 @leesf : Can you folks loop in someone to assist here please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4698: [SUPPORT] RowDataToAvroConverters does not support data in flink timestamp_ltz (timestamp_with_local_time_zone) format.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4698:
URL: https://github.com/apache/hudi/issues/4698#issuecomment-1039622562
thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hanjun996 commented on issue #4698: [SUPPORT] RowDataToAvroConverters does not support data in flink timestamp_ltz (timestamp_with_local_time_zone) format.
Posted by GitBox <gi...@apache.org>.
hanjun996 commented on issue #4698:
URL: https://github.com/apache/hudi/issues/4698#issuecomment-1022904157
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org