[python] Unknown *and* invalid logical types

[Ryan Skraba <ry...@skraba.com>]

Hello!  For context, I've been looking to brush up on my python, and
picked a simple-looking JIRA for practice :D  Nothing is as simple as
it seems!

AVRO-2429 asks to not throw exceptions when a LogicalType is unknown,
falling back to the primitive type.  The spec adds that an invalid
logical type should also fall back (e.g. a decimal with badly
configured precision).

The Java implementation ignores unknown and invalid types (tested with
the SpecificCompiler.

The Python implementation currently throws an exception when an
invalid logical type is encountered.  I wrote a PR to change the
behaviour to fall back to the primitive type:
https://github.com/apache/avro/pull/687

Although this conforms to the spec, it's probably not *ideal* for
python: unlike Java code using a specific record, an expected datum
due to a bad logical type in the schema won't be detected at compile
time...

Michael A. Smith suggested making this a configurable warning so the
user can control whether the misconfiguration is shown, ignored or
causes failure -- sound fine to everyone?

Best regards, Ryan

Re: [python] Unknown *and* invalid logical types

["Michael A. Smith" <mi...@smith-li.com>]

What I'm proposing is that we use the native python warnings module:
https://docs.python.org/2/library/warnings.html

> Warning messages are normally written to sys.stderr, but their
disposition can be changed flexibly, from ignoring all warnings to turning
them into exceptions. The disposition of warnings can vary based on the
warning category (see below), the text of the warning message, and the
source location where it is issued. Repetitions of a particular warning for
the same source location are typically suppressed.

The module also supports easily transforming warnings into logger.warn
calls.

I think we should create a subclass of UserWarning for Avro warnings from
which all future avro warnings should extend, and then create an
IgnoredLogicalType warning from that.

I think the warning should be enabled in "once" mode by default.

I'm curious what others' opinions are.

On Thu, Oct 31, 2019 at 06:09 Ryan Skraba <ry...@skraba.com> wrote:

> Hello!  For context, I've been looking to brush up on my python, and
> picked a simple-looking JIRA for practice :D  Nothing is as simple as
> it seems!
>
> AVRO-2429 asks to not throw exceptions when a LogicalType is unknown,
> falling back to the primitive type.  The spec adds that an invalid
> logical type should also fall back (e.g. a decimal with badly
> configured precision).
>
> The Java implementation ignores unknown and invalid types (tested with
> the SpecificCompiler.
>
> The Python implementation currently throws an exception when an
> invalid logical type is encountered.  I wrote a PR to change the
> behaviour to fall back to the primitive type:
> https://github.com/apache/avro/pull/687
>
> Although this conforms to the spec, it's probably not *ideal* for
> python: unlike Java code using a specific record, an expected datum
> due to a bad logical type in the schema won't be detected at compile
> time...
>
> Michael A. Smith suggested making this a configurable warning so the
> user can control whether the misconfiguration is shown, ignored or
> causes failure -- sound fine to everyone?
>
> Best regards, Ryan
>