Support for Microseconds in Avro Deserialization

[Joseph Lorenzini <JL...@gohealth.com>]

Hi all,



The avro specification supports microseconds and reviewing the source code in
org.apache.avro.LogicalTypes seems to indicate microsecond support. However,
the conversion code in flink (see
org.apache.flink.formats.avro.typeutils.AvroSchemaConverter#convertToSchema)
has this checked:



                if (precision <= 3) {

                    avroLogicalType = LogicalTypes.timestampMillis();

                } else {

                    throw new IllegalArgumentException(

                            "Avro does not support TIMESTAMP type "

                                    + "with precision: "

                                    + precision

                                    + ", it only supports precision less than 3.");

                }



So it seems that flink only supports managing avro timestamps with at most
millisecond precision. Does someone have a brief explanation about why this
limitation exists? Depending on how complicated it is, I’d be willing to
submit a PR to add that support in.



Thanks,

Joe

Privileged/Confidential Information may be contained in this message. If you
are not the addressee indicated in this message (or responsible for delivery
of the message to such person), you may not copy or deliver this message to
anyone. In such case, you should destroy this message and kindly notify the
sender by reply email. Please advise immediately if you or your employer does
not consent to Internet email for messages of this kind. Opinions, conclusions
and other information in this message that do not relate to the official
business of my firm shall be understood as neither given nor endorsed by it.

Re: Support for Microseconds in Avro Deserialization

[Robert Metzger <rm...@apache.org>]

Hey Joe,
thanks a lot for reaching out regarding this.
I have no explanation for why this exists, but since there's not ticket
about this yet, I filed one:
https://issues.apache.org/jira/browse/FLINK-23589
I also pinged some committers who can hopefully provide some
additional context.

I would propose to continue the discussion in Jira!

Thanks again!


On Mon, Aug 2, 2021 at 3:17 PM Joseph Lorenzini <JL...@gohealth.com>
wrote:

> Hi all,
>
>
>
> The avro specification supports microseconds and reviewing the source code
> in org.apache.avro.LogicalTypes seems to indicate microsecond support.
> However, the conversion code in flink (see
> org.apache.flink.formats.avro.typeutils.AvroSchemaConverter#convertToSchema)
> has this checked:
>
>
>
>                 if (precision <= 3) {
>
>                     avroLogicalType = LogicalTypes.timestampMillis();
>
>                 } else {
>
>                     throw new IllegalArgumentException(
>
>                             "Avro does not support TIMESTAMP type "
>
>                                     + "with precision: "
>
>                                     + precision
>
>                                     + ", it only supports precision less
> than 3.");
>
>                 }
>
>
>
> So it seems that flink only supports managing avro timestamps with at most
> millisecond precision. Does someone have a brief explanation about why this
> limitation exists? Depending on how complicated it is, I’d be willing to
> submit a PR to add that support in.
>
>
>
> Thanks,
>
> Joe
> Privileged/Confidential Information may be contained in this message. If
> you are not the addressee indicated in this message (or responsible for
> delivery of the message to such person), you may not copy or deliver this
> message to anyone. In such case, you should destroy this message and kindly
> notify the sender by reply email. Please advise immediately if you or your
> employer does not consent to Internet email for messages of this kind.
> Opinions, conclusions and other information in this message that do not
> relate to the official business of my firm shall be understood as neither
> given nor endorsed by it.
>