clarification on schema resolution

[Scott Banachowski <sb...@yahoo-inc.com>]

Hi,
These questions might be obvious, but I want to make sure, just in case.

Regarding union resolution, the spec says to take the first match when
comparing them.  What if the first match is "promotable"?  I would expect
preference for the first "exact" match, if both an exact and promotable
match existed.

example, my writer puts a "long" and my reader has union:
[ double long ]

The first match is double (due to promotion), but in this case I'd probably
want to use long, correct?

but if my reader is:
[ double int ] 

Is it ok to settle for conversion to double?  Or do we only allow exact
matches in union comparisons?

Second clarification is about usage of unset vs. signaling error:  in the
rules if we cannot match enums, we declare it unset, but if we cannot match
unions, we signal error.  It seems for one type we take a best effort
approach and in the other we give up.  I'm just trying to understand what is
different about the two cases.  And assuming we're doing this matching at
runtime, for example, reading an rpc message, does signaling error mean we
drop the message, or still pass parts of the message that could be resolved?

thanks,
Scott

 

Re: clarification on schema resolution

[Jeff Hammerbacher <ha...@cloudera.com>]

> What do folks think is best here?  Should we require all values that don't
> have a default values to be present, i.e., record fields with a default
> value are optional, all others are mandatory?
>

Thrift allows entries in a struct to be marked "optional", which may be an
option to make explicit (and conflict with default), though they note it
complicates certain language bindings.

Later,
Jeff

Re: clarification on schema resolution

[Scott Carey <sc...@richrelevance.com>]

On 11/18/09 1:51 PM, "Doug Cutting" <cu...@apache.org> wrote:

> I don't know that these conventions are always the best.  Consider the
> extreme: if the reader's top-level schema is "int" and the writer wrote
> using a "float" schema, should we:
> 
>   a. require that implementations signal an error;
>   b. require that implementations ignore this, e.g., returning null or
> unset or somesuch; or,
>   c. permit implementations to vary?
> 
> Currently the spec indicates (a). This applies equally to field type
> mismatches as to top-level type mismatches.  That said, there might be
> cases where an application wants to read all fields it can map into an
> existing datastructure, ignoring whatever it cannot read.
> 
> Thus we might change the spec to say that implementations are
> recommended (but not required) to signal errors in such cases.  An
> implementation that signals such errors can state in its documentation
> that it implements the recommended resolution algorithm.  An
> implementation that does not should state how it deviates.
> 

This strikes me as something that is going to be very application/user
specific.

Maybe, this is something a client can specify in the schema? For field name
"foo" that the client thinks should be "int",  "promotefrom" = "float" might
signal that its ok to cast the float to an int without error.  We could have
a set of universal defaults with configurable overrides for each type.  We
already have "default" to deal with missing items, what to do with
mismatches seems to be along similar lines.

Such a configuration could live outside the schema in some sort of client
configuration.  Although this again could map to annotations in the java
case, something that is general and applies to all languages might be
better.  Thus, the schema itself, or metadata around it / within it.

Specifically, I'm starting to think ahead to the challenges of schema
migration and versioning.  If I'm going to start using Avro (hopefully in 3
weeks!) I need to think about and solve my approach here.  Although a good
set of default behavior is mandatory, I can't help but think that I'll run
into a situation that I want to override that behavior.



> 
> Doug
> 

Re: clarification on schema resolution

[Doug Cutting <cu...@apache.org>]

Scott Banachowski wrote:
> What I am really wondering is what flagging an error means in this context.
> What if this is a field of a record--in this case, does error mean that the
> entire record is bad?  Or do I simply ignore this field and process the rest
> of the record?  And if I do the latter, how is this different than "unset"?

The entire record is bad.  We leave a field unset when the written 
record has no field with that name.  But when it does have a field with 
that name but with a different type, this is a type mismatch and is 
specified to be an error.

I don't know that these conventions are always the best.  Consider the 
extreme: if the reader's top-level schema is "int" and the writer wrote 
using a "float" schema, should we:

  a. require that implementations signal an error;
  b. require that implementations ignore this, e.g., returning null or 
unset or somesuch; or,
  c. permit implementations to vary?

Currently the spec indicates (a). This applies equally to field type 
mismatches as to top-level type mismatches.  That said, there might be 
cases where an application wants to read all fields it can map into an 
existing datastructure, ignoring whatever it cannot read.

Thus we might change the spec to say that implementations are 
recommended (but not required) to signal errors in such cases.  An 
implementation that signals such errors can state in its documentation 
that it implements the recommended resolution algorithm.  An 
implementation that does not should state how it deviates.

Doug

Re: clarification on schema resolution

[Scott Banachowski <sb...@yahoo-inc.com>]

On 11/16/09 11:24 AM, "Doug Cutting" said:

> Scott Banachowski wrote:
>> Regarding union resolution, the spec says to take the first match when
>> comparing them.  What if the first match is "promotable"?  I would expect
>> preference for the first "exact" match, if both an exact and promotable
>> match existed.
> 
> Good question.  I agree, and that's what the Java implementation does,
> in GenericDatumReader#resolveExpected().  It first looks for a branch
> which "exactly" matches the type written, where "exactly" means, for
> named types, has the same name, and for other types has the same base
> type, e.g., array, map, int.  If exact match fails, then it looks for a
> match via promotion.
> 
> Should we fix the spec to better document this?

Yes, I think the spec can say something along the lines of, first try to
find an exact match, and if not present, use the first match via promotion.

> 
>> Second clarification is about usage of unset vs. signaling error:  in the
>> rules if we cannot match enums, we declare it unset, but if we cannot match
>> unions, we signal error.  It seems for one type we take a best effort
>> approach and in the other we give up.  I'm just trying to understand what is
>> different about the two cases.
> 
> I think it's reasonable to differ in how we handle unmatched field names
> and mismatched types.  If the type for a field changes in a way that
> cannot be promoted, that's an incompatible schema.  But if a field is
> renamed, then we ignore the value under the old name and might leave the
> new name unset.  Union mismatches are type differences, not field name
> differences, so errors seem appropriate.  Does that make sense?

It makes sense.  But let me ask this question in the context of a specific
example:

Let's say my writer is "[int double]" and my reader is "int".
If the writer sets the value as an int, obviously my reader can proceed
without error.   But if the writer sets the value as double, my reader
should flag an error.

What I am really wondering is what flagging an error means in this context.
What if this is a field of a record--in this case, does error mean that the
entire record is bad?  Or do I simply ignore this field and process the rest
of the record?  And if I do the latter, how is this different than "unset"?

thanks,
Scott






> 
> There's an open issue about whether leaving fields unset is reasonable.
> 
> https://issues.apache.org/jira/browse/AVRO-75
> 
> What do folks think is best here?  Should we require all values that
> don't have a default values to be present, i.e., record fields with a
> default value are optional, all others are mandatory?
> 
>> And assuming we're doing this matching at
>> runtime, for example, reading an rpc message, does signaling error mean we
>> drop the message, or still pass parts of the message that could be resolved?
> 
> If we adopt the "unset" concept, then implementations could pass the
> parts of the message that can be resolved.  Note that we should never
> simply drop the message, but rather return an error response, but that's
> probably what you meant.
> 
> Doug

Re: clarification on schema resolution

[Doug Cutting <cu...@apache.org>]

Scott Banachowski wrote:
> Regarding union resolution, the spec says to take the first match when
> comparing them.  What if the first match is "promotable"?  I would expect
> preference for the first "exact" match, if both an exact and promotable
> match existed.

Good question.  I agree, and that's what the Java implementation does, 
in GenericDatumReader#resolveExpected().  It first looks for a branch 
which "exactly" matches the type written, where "exactly" means, for 
named types, has the same name, and for other types has the same base 
type, e.g., array, map, int.  If exact match fails, then it looks for a 
match via promotion.

Should we fix the spec to better document this?

> Second clarification is about usage of unset vs. signaling error:  in the
> rules if we cannot match enums, we declare it unset, but if we cannot match
> unions, we signal error.  It seems for one type we take a best effort
> approach and in the other we give up.  I'm just trying to understand what is
> different about the two cases.

I think it's reasonable to differ in how we handle unmatched field names 
and mismatched types.  If the type for a field changes in a way that 
cannot be promoted, that's an incompatible schema.  But if a field is 
renamed, then we ignore the value under the old name and might leave the 
new name unset.  Union mismatches are type differences, not field name 
differences, so errors seem appropriate.  Does that make sense?

There's an open issue about whether leaving fields unset is reasonable.

https://issues.apache.org/jira/browse/AVRO-75

What do folks think is best here?  Should we require all values that 
don't have a default values to be present, i.e., record fields with a 
default value are optional, all others are mandatory?

> And assuming we're doing this matching at
> runtime, for example, reading an rpc message, does signaling error mean we
> drop the message, or still pass parts of the message that could be resolved?

If we adopt the "unset" concept, then implementations could pass the 
parts of the message that can be resolved.  Note that we should never 
simply drop the message, but rather return an error response, but that's 
probably what you meant.

Doug