Re: Avro schema resolution

[Sean Busbey <bu...@cloudera.com>]

What's the rationale for not allowing it at all? Given that it's been in
the spec so long I'm hesitant to remove it.

I can't think of a time that we did a breaking change to the spec. Even if
we check all of our implementations, there's no way to know if someone else
has already created an implementation that allows the promotion since it's
in the spec.

On Wed, Aug 30, 2017 at 2:45 AM, Nandor Kollar <nk...@cloudera.com> wrote:

> Thanks Doug! In this case, I think in this case ResolvingGrammarGenerator
> doesn't implement the schema resolution logic according to the spec,
> created a Jira for it:
>
> https://issues.apache.org/jira/browse/AVRO-2072
>
> What do you think about Anders comment about to int -> float promotion
> on AVRO-1931?
> Should Avro allow it at all?
>
> Nandor
>
> On Wed, Aug 30, 2017 at 1:53 AM, Doug Cutting <cu...@gmail.com> wrote:
>
> > I'm not sure why both SchemaCompatibility and SchemaValidator exist.  I
> > think folks perhaps just didn't realize the former existed when the
> latter
> > was added.
> >
> > https://issues.apache.org/jira/browse/AVRO-1409
> > https://issues.apache.org/jira/browse/AVRO-1315
> >
> > The latter looks like a probably more efficient implementation, since it
> > doesn't compile a parser for each schema validated, but the former shares
> > logic with the reading & writing of data, simplifying maintenance.
> >
> > Doug
> >
> > On Wed, Aug 23, 2017 at 6:01 AM, Nandor Kollar <nk...@cloudera.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > While I was trying to understand how schema evolution works in Avro, I
> > > noticed that the schema resolution logic (
> > > https://avro.apache.org/docs/1.8.1/spec.html#Schema+Resolution) is
> > > implemented in two different places in Avro: in
> ResolvingGrammarGenerator
> > > and in SchemaCompatibility.
> > >
> > > Could someone please explain the differences between
> > > ResolvingGrammarGenerator, SchemaCompatibility, and SchemaValidator? It
> > > seems that SchemaValidator is used to check the compatibility between
> the
> > > reader and the writer schema, which sounds exactly what
> > SchemaCompatibility
> > > does, why does it use ResolvingGrammarGenerator instead of
> > > SchemaCompatibility? Why is the same schema resolution logic
> implemented
> > > twice (if I understood correctly)?
> > >
> > > I also have a question related to unions and schema resolution. In the
> > > patch for AVRO-1933 Anders commented out a case for unions: union with
> an
> > > int branch is not compatible with a long union reader. After a brief
> > > investigation I noticed, that ResolvingGrammarGenerator#bestBranch
> > doesn't
> > > allow int -> float promotion in union branches. Is this a bug? Should
> we
> > > allow this promotion at all, since like Anders mentioned in AVRO-1931
> > this
> > > is a lossy conversion? What do you think?
> > >
> > > Thanks,
> > > Nandor
> > >
> >
>



-- 
busbey

Re: Avro schema resolution

[Doug Cutting <cu...@gmail.com>]

This seems like a bug in the Java implementation, not a mistake in the
spec.  Int->float promotion is supported elsewhere and, while lossy, has
always been in the spec.  There are other lossy promotions (e.g.,
long->float) and dropping missing record fields is lossy, so lossiness
alone doesn't seem a sufficient criterion to disallow this promotion.

Doug

On Fri, Sep 1, 2017 at 6:32 AM, Sean Busbey <bu...@cloudera.com> wrote:

> What's the rationale for not allowing it at all? Given that it's been in
> the spec so long I'm hesitant to remove it.
>
> I can't think of a time that we did a breaking change to the spec. Even if
> we check all of our implementations, there's no way to know if someone else
> has already created an implementation that allows the promotion since it's
> in the spec.
>
> On Wed, Aug 30, 2017 at 2:45 AM, Nandor Kollar <nk...@cloudera.com>
> wrote:
>
> > Thanks Doug! In this case, I think in this case ResolvingGrammarGenerator
> > doesn't implement the schema resolution logic according to the spec,
> > created a Jira for it:
> >
> > https://issues.apache.org/jira/browse/AVRO-2072
> >
> > What do you think about Anders comment about to int -> float promotion
> > on AVRO-1931?
> > Should Avro allow it at all?
> >
> > Nandor
> >
> > On Wed, Aug 30, 2017 at 1:53 AM, Doug Cutting <cu...@gmail.com> wrote:
> >
> > > I'm not sure why both SchemaCompatibility and SchemaValidator exist.  I
> > > think folks perhaps just didn't realize the former existed when the
> > latter
> > > was added.
> > >
> > > https://issues.apache.org/jira/browse/AVRO-1409
> > > https://issues.apache.org/jira/browse/AVRO-1315
> > >
> > > The latter looks like a probably more efficient implementation, since
> it
> > > doesn't compile a parser for each schema validated, but the former
> shares
> > > logic with the reading & writing of data, simplifying maintenance.
> > >
> > > Doug
> > >
> > > On Wed, Aug 23, 2017 at 6:01 AM, Nandor Kollar <nk...@cloudera.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > While I was trying to understand how schema evolution works in Avro,
> I
> > > > noticed that the schema resolution logic (
> > > > https://avro.apache.org/docs/1.8.1/spec.html#Schema+Resolution) is
> > > > implemented in two different places in Avro: in
> > ResolvingGrammarGenerator
> > > > and in SchemaCompatibility.
> > > >
> > > > Could someone please explain the differences between
> > > > ResolvingGrammarGenerator, SchemaCompatibility, and SchemaValidator?
> It
> > > > seems that SchemaValidator is used to check the compatibility between
> > the
> > > > reader and the writer schema, which sounds exactly what
> > > SchemaCompatibility
> > > > does, why does it use ResolvingGrammarGenerator instead of
> > > > SchemaCompatibility? Why is the same schema resolution logic
> > implemented
> > > > twice (if I understood correctly)?
> > > >
> > > > I also have a question related to unions and schema resolution. In
> the
> > > > patch for AVRO-1933 Anders commented out a case for unions: union
> with
> > an
> > > > int branch is not compatible with a long union reader. After a brief
> > > > investigation I noticed, that ResolvingGrammarGenerator#bestBranch
> > > doesn't
> > > > allow int -> float promotion in union branches. Is this a bug? Should
> > we
> > > > allow this promotion at all, since like Anders mentioned in AVRO-1931
> > > this
> > > > is a lossy conversion? What do you think?
> > > >
> > > > Thanks,
> > > > Nandor
> > > >
> > >
> >
>
>
>
> --
> busbey
>