You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by Raymie Stata <rs...@yahoo.com.INVALID> on 2018/10/24 06:33:43 UTC
Resolving Enumerations -- bug?
While working on AVRO-2090, I noticed what is either an implementation
bug or a specification bug in schema resolution for enumerations.
The relevant code is here: https://bit.ly/2q5tsIp. This code uses the
reader's default symbol, if it exists, in the case where the writer's
symbols is missing.
Let's think about this through an example. Let's say the reader
defines just two symbols for an enum: "alpha" and "beta", with "one"
as the default. Let's say that the writer had three symbols: "alpha",
"beta", and "gamma". The way https://bit.ly/2q5tsIp is written, if
the reader encounters a file containing the symbol "gamma", and error
will NOT be thrown. Instead, the reader will be told that the actual
symbol was "alpha".
Note that the Avro specification says the following about matching
enumerations: "if the writer's symbol is not present in the reader's
enum, then an error is signalled." This would suggest that, in the
example just described, an error should be thrown, rather than the
value "alpha" returns. So either the code is wrong, or the spec is
wrong.
On a related note, the current spec says nothing about a "default"
property for enumerations. When should this property be used? As a
"default default" for fields? (If so, this isn't happening.) As a
value to be used in resolution, when the writer provides a symbole
that is not (any longer) defined? (If so, this is happening in the
code, but the spec needs an update.) And/or should it be used in
other circumstances?
I'm willing to update docs and/or code appropriately, but can someone
indicated the intended semantics of "default" for enums?
Raymie
Re: Resolving Enumerations -- bug?
Posted by Raymie Stata <rs...@yahoo.com.INVALID>.
Just posted a pull request for an update to the spec doc:
https://github.com/apache/avro/pull/357
On Thu, Oct 25, 2018 at 10:06 AM Doug Cutting <cu...@gmail.com> wrote:
>
> This is https://issues.apache.org/jira/browse/AVRO-1340. Caution, it's a
> long discussion!
>
> In your example, I think you mean that the default is "alpha", not "one".
>
> It looks like 1340 neglected to update the spec. My mistake for missing
> that in review.
>
> The idea is to permit adding/removing enum symbols. So this is different
> from field defaults, which deal with adding/removing fields.
>
> Thanks for catching this.
>
> Doug
>
>
>
>
> On Tue, Oct 23, 2018 at 11:34 PM Raymie Stata <rs...@yahoo.com.invalid>
> wrote:
>
> > While working on AVRO-2090, I noticed what is either an implementation
> > bug or a specification bug in schema resolution for enumerations.
> >
> > The relevant code is here: https://bit.ly/2q5tsIp. This code uses the
> > reader's default symbol, if it exists, in the case where the writer's
> > symbols is missing.
> >
> > Let's think about this through an example. Let's say the reader
> > defines just two symbols for an enum: "alpha" and "beta", with "one"
> > as the default. Let's say that the writer had three symbols: "alpha",
> > "beta", and "gamma". The way https://bit.ly/2q5tsIp is written, if
> > the reader encounters a file containing the symbol "gamma", and error
> > will NOT be thrown. Instead, the reader will be told that the actual
> > symbol was "alpha".
> >
> > Note that the Avro specification says the following about matching
> > enumerations: "if the writer's symbol is not present in the reader's
> > enum, then an error is signalled." This would suggest that, in the
> > example just described, an error should be thrown, rather than the
> > value "alpha" returns. So either the code is wrong, or the spec is
> > wrong.
> >
> > On a related note, the current spec says nothing about a "default"
> > property for enumerations. When should this property be used? As a
> > "default default" for fields? (If so, this isn't happening.) As a
> > value to be used in resolution, when the writer provides a symbole
> > that is not (any longer) defined? (If so, this is happening in the
> > code, but the spec needs an update.) And/or should it be used in
> > other circumstances?
> >
> > I'm willing to update docs and/or code appropriately, but can someone
> > indicated the intended semantics of "default" for enums?
> >
> > Raymie
> >
Re: Resolving Enumerations -- bug?
Posted by Doug Cutting <cu...@gmail.com>.
This is https://issues.apache.org/jira/browse/AVRO-1340. Caution, it's a
long discussion!
In your example, I think you mean that the default is "alpha", not "one".
It looks like 1340 neglected to update the spec. My mistake for missing
that in review.
The idea is to permit adding/removing enum symbols. So this is different
from field defaults, which deal with adding/removing fields.
Thanks for catching this.
Doug
On Tue, Oct 23, 2018 at 11:34 PM Raymie Stata <rs...@yahoo.com.invalid>
wrote:
> While working on AVRO-2090, I noticed what is either an implementation
> bug or a specification bug in schema resolution for enumerations.
>
> The relevant code is here: https://bit.ly/2q5tsIp. This code uses the
> reader's default symbol, if it exists, in the case where the writer's
> symbols is missing.
>
> Let's think about this through an example. Let's say the reader
> defines just two symbols for an enum: "alpha" and "beta", with "one"
> as the default. Let's say that the writer had three symbols: "alpha",
> "beta", and "gamma". The way https://bit.ly/2q5tsIp is written, if
> the reader encounters a file containing the symbol "gamma", and error
> will NOT be thrown. Instead, the reader will be told that the actual
> symbol was "alpha".
>
> Note that the Avro specification says the following about matching
> enumerations: "if the writer's symbol is not present in the reader's
> enum, then an error is signalled." This would suggest that, in the
> example just described, an error should be thrown, rather than the
> value "alpha" returns. So either the code is wrong, or the spec is
> wrong.
>
> On a related note, the current spec says nothing about a "default"
> property for enumerations. When should this property be used? As a
> "default default" for fields? (If so, this isn't happening.) As a
> value to be used in resolution, when the writer provides a symbole
> that is not (any longer) defined? (If so, this is happening in the
> code, but the spec needs an update.) And/or should it be used in
> other circumstances?
>
> I'm willing to update docs and/or code appropriately, but can someone
> indicated the intended semantics of "default" for enums?
>
> Raymie
>
Re: Resolving Enumerations -- bug?
Posted by Raymie Stata <rs...@yahoo.com.INVALID>.
Anybody?
On Tue, Oct 23, 2018 at 11:33 PM Raymie Stata <rs...@yahoo.com> wrote:
>
> While working on AVRO-2090, I noticed what is either an implementation
> bug or a specification bug in schema resolution for enumerations.
>
> The relevant code is here: https://bit.ly/2q5tsIp. This code uses the
> reader's default symbol, if it exists, in the case where the writer's
> symbols is missing.
>
> Let's think about this through an example. Let's say the reader
> defines just two symbols for an enum: "alpha" and "beta", with "one"
> as the default. Let's say that the writer had three symbols: "alpha",
> "beta", and "gamma". The way https://bit.ly/2q5tsIp is written, if
> the reader encounters a file containing the symbol "gamma", and error
> will NOT be thrown. Instead, the reader will be told that the actual
> symbol was "alpha".
>
> Note that the Avro specification says the following about matching
> enumerations: "if the writer's symbol is not present in the reader's
> enum, then an error is signalled." This would suggest that, in the
> example just described, an error should be thrown, rather than the
> value "alpha" returns. So either the code is wrong, or the spec is
> wrong.
>
> On a related note, the current spec says nothing about a "default"
> property for enumerations. When should this property be used? As a
> "default default" for fields? (If so, this isn't happening.) As a
> value to be used in resolution, when the writer provides a symbole
> that is not (any longer) defined? (If so, this is happening in the
> code, but the spec needs an update.) And/or should it be used in
> other circumstances?
>
> I'm willing to update docs and/or code appropriately, but can someone
> indicated the intended semantics of "default" for enums?
>
> Raymie