You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Y Ethan Guo (Jira)" <ji...@apache.org> on 2024/10/09 01:59:00 UTC
[jira] [Updated] (HUDI-7985) Support more formats in timestamp logical types in Json Avro converter
[ https://issues.apache.org/jira/browse/HUDI-7985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Y Ethan Guo updated HUDI-7985:
------------------------------
Sprint: (was: Hudi 1.0 Sprint2024/10/7-10/13)
> Support more formats in timestamp logical types in Json Avro converter
> ----------------------------------------------------------------------
>
> Key: HUDI-7985
> URL: https://issues.apache.org/jira/browse/HUDI-7985
> Project: Apache Hudi
> Issue Type: Improvement
> Reporter: Ethan Guo (this is the old account; please use "yihua")
> Assignee: Y Ethan Guo
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.0.0
>
>
> Following error is thrown when using Json Kafka Source with transformer and decimal is in the schema:
> {code:java}
> Caused by: Json to Avro Type conversion error for field loaded_at, 2024-06-03 13:42:34.951+00:00 for {"type":"long","logicalType":"timestamp-millis"}
> at org.apache.hudi.avro.MercifulJsonConverter$JsonToAvroFieldProcessorUtil$JsonToAvroFieldProcessor.convertToAvro(MercifulJsonConverter.java:194)
> at org.apache.hudi.avro.MercifulJsonConverter$JsonToAvroFieldProcessorUtil.convertToAvro(MercifulJsonConverter.java:204)
> at org.apache.hudi.avro.MercifulJsonConverter.convertJsonToAvroField(MercifulJsonConverter.java:182)
> at org.apache.hudi.avro.MercifulJsonConverter.convertJsonToAvro(MercifulJsonConverter.java:126)
> at org.apache.hudi.avro.MercifulJsonConverter.convert(MercifulJsonConverter.java:107)
> at org.apache.hudi.utilities.sources.helpers.AvroConvertor.fromJson(AvroConvertor.java:118)
> ... 43 more {code}
> We need to make sure "2024-06-03 13:42:34.951+00:00" is supported in timestamp logical type.
> * ISO 8601 supports the zone offset in the standard, e.g., {{+01:00}} , and {{Z}} is the zone offset equivalent to {{+00:00}} or UTC ([ref1|https://en.wikipedia.org/wiki/ISO_8601#Time_zone_designators])
> * {{2011-12-03T10:15:30+01:00}} conforms to ISO 8601 with {{T}} as the separation character
> * There are systems that use \{{ }} (space) instead of {{T}} as the separation (other parts are the same). References indicate that ISO-8601 used to allow this by _mutual agreement_ ([ref2|https://stackoverflow.com/questions/30201003/how-to-deal-with-optional-t-in-iso-8601-timestamp-in-java-8-jsr-310-threet], [ref3|https://www.reddit.com/r/ISO8601/comments/173r61j/t_vs_space_separation_of_date_and_time/])
> * {{DateTimeFormatter.ISO_OFFSET_DATE_TIME}} can successfully parse timestamps like {{2024-05-13T23:53:36.004Z}} , already supported in {{{}MercifulJsonConverter{}}}, and additionally {{2011-12-03T10:15:30+01:00}} with zone offset (which is not supported in {{MercifulJsonConverter}} yet)
> * {{DateTimeFormatter.ISO_OFFSET_DATE_TIME}} cannot parse the timestamp with space as the separator, like {{2011-12-03 10:15:30+01:00}} . But with a simple twist of the formatter, it can be easily supported.
> My take is we should change the formatter of the timestamp logical types to support zone offset and space character as the separator (which is backwards compatible), instead of introducing a new config of format (assuming that common use cases just have space character as the variant).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)