You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Christophe Taton <ta...@wibidata.com> on 2013/07/14 04:01:30 UTC
Properties on union schema?
Hi,
I'm toying with a few changes to provide alternative representations of
union fields in Java (somewhat related to
https://issues.apache.org/jira/browse/AVRO-248).
To experiment with this, I'd like to set properties on union schemas, but
properties are currently disabled on unions.
Is there a particular reason for this, or is it a reasonable change to
allow properties on unions?
Thanks,
C.
Re: Properties on union schema?
Posted by Christophe Taton <ta...@wibidata.com>.
Hi Connor,
Wrapping unions in records is possible, but appears very inconvenient in
several ways (both from the user's and the implementation's perspective).
Union is the only schema type for which properties are turned off, and I
cannot find any documentation or reason for it.
C.
On Sat, Jul 13, 2013 at 7:15 PM, Connor Doyle <co...@gmail.com> wrote:
> Perhaps I'm missing part of the original intent for this enhancement, but
> couldn't users define a record to wrap a single union type if so desired?
> The (binary) encoding would be identical.
> --
> Connor
>
> On Jul 13, 2013, at 21:01, Christophe Taton <ta...@wibidata.com> wrote:
>
> Hi,
> I'm toying with a few changes to provide alternative representations of
> union fields in Java (somewhat related to
> https://issues.apache.org/jira/browse/AVRO-248).
> To experiment with this, I'd like to set properties on union schemas, but
> properties are currently disabled on unions.
> Is there a particular reason for this, or is it a reasonable change to
> allow properties on unions?
> Thanks,
> C.
>
>
Re: Properties on union schema?
Posted by Connor Doyle <co...@gmail.com>.
Perhaps I'm missing part of the original intent for this enhancement, but couldn't users define a record to wrap a single union type if so desired? The (binary) encoding would be identical.
--
Connor
On Jul 13, 2013, at 21:01, Christophe Taton <ta...@wibidata.com> wrote:
> Hi,
> I'm toying with a few changes to provide alternative representations of union fields in Java (somewhat related to https://issues.apache.org/jira/browse/AVRO-248).
> To experiment with this, I'd like to set properties on union schemas, but properties are currently disabled on unions.
> Is there a particular reason for this, or is it a reasonable change to allow properties on unions?
> Thanks,
> C.
Re: Properties on union schema?
Posted by Doug Cutting <cu...@apache.org>.
A concern is that folks might write data, then later find they cannot
read it using tools they might otherwise expect to read it with. They
might not initially use a feature that triggers the incompatibility,
only to find it later, when they've already committed to the newer
version.
To reduce the likelihood of such scenarios we should alter the file's
magic number when incompatible changes are included. We can also
increment the project and specification's major version number. If
such changes are optional then we might only increment the file's
magic number when they're enabled. Then the project's major version
number might not be incremented until these are made default.
Since there are multiple implementations of Avro it behooves us not to
introduce too many magic numbers that will need long-term support.
(Once a format is used in production, it probably needs to be
supported forever.) Perhaps we should start considering a list of
changes to be made in Avro 2.0's schema language. We might use an
"experimental" magic number during development that prints obnoxious
warnings when features are enabled so that folks are well warned not
to use them in production until the full set of new features is
determined.
What do others think?
Doug
On Thu, Jul 18, 2013 at 10:23 AM, Christophe Taton <ta...@wibidata.com> wrote:
> Hi Doug,
>
> I realize I have strong interests in several issues that would imply
> forward-incompatible changes (eg. AVRO-248 and AVRO-530).
> Do you believe it is possible to introduce part of these as experimental
> features that would be disabled by default, or would you rather isolate such
> changes in completely separate releases? I would happily invest energy in
> either approach.
>
> Thanks,
> C.
>
>
> On Tue, Jul 16, 2013 at 11:13 AM, Doug Cutting <cu...@apache.org> wrote:
>>
>> We've promised to maintain forward and backward compatibility until
>> version 2.0. There have been a number of incompatible schema changes
>> requested over the history of Avro, but none have yet been made. If
>> we ever decide to make any, we should perhaps try to make them all at
>> once and provide good conversion tools.
>>
>> Doug
>>
>> On Tue, Jul 16, 2013 at 12:40 AM, Christophe Taton <ta...@wibidata.com>
>> wrote:
>> > Hi Doug,
>> >
>> > Thanks for the explanation.
>> > How far back is it reasonable to maintain backward and/or forward
>> > compatibility?
>> >
>> > Assuming there is a limit to the forward compatibility requirement,
>> > could we
>> > introduce the ability to read schemas with extended union descriptors,
>> > without the ability to write such descriptors, and introduce the ability
>> > to
>> > write after enough releases have passed?
>> >
>> > Thanks,
>> > C.
>> >
>> >
>> > On Mon, Jul 15, 2013 at 11:25 AM, Doug Cutting <cu...@apache.org>
>> > wrote:
>> >>
>> >> The reason is simply that the JSON syntax for union schemas doesn't
>> >> permit properties. To support properties we'd need to add an
>> >> alternate syntax for unions, but I don't see how to do that
>> >> compatibly, so that an old client seeing one of the new union schemas
>> >> would still be able to process the data.
>> >>
>> >> For example, the syntax might be something like:
>> >>
>> >> {"type":"union", "branches":[...], "prop1":"val1", ...}
>> >>
>> >> But that would cause errors in every existing implementation.
>> >>
>> >> Doug
>> >>
>> >> On Sat, Jul 13, 2013 at 7:01 PM, Christophe Taton <ta...@wibidata.com>
>> >> wrote:
>> >> > Hi,
>> >> > I'm toying with a few changes to provide alternative representations
>> >> > of
>> >> > union fields in Java (somewhat related to
>> >> > https://issues.apache.org/jira/browse/AVRO-248).
>> >> > To experiment with this, I'd like to set properties on union schemas,
>> >> > but
>> >> > properties are currently disabled on unions.
>> >> > Is there a particular reason for this, or is it a reasonable change
>> >> > to
>> >> > allow
>> >> > properties on unions?
>> >> > Thanks,
>> >> > C.
>> >
>> >
>
>
Re: Properties on union schema?
Posted by Christophe Taton <ta...@wibidata.com>.
Hi Doug,
I realize I have strong interests in several issues that would imply
forward-incompatible changes (eg. AVRO-248 and AVRO-530).
Do you believe it is possible to introduce part of these as experimental
features that would be disabled by default, or would you rather isolate
such changes in completely separate releases? I would happily invest energy
in either approach.
Thanks,
C.
On Tue, Jul 16, 2013 at 11:13 AM, Doug Cutting <cu...@apache.org> wrote:
> We've promised to maintain forward and backward compatibility until
> version 2.0. There have been a number of incompatible schema changes
> requested over the history of Avro, but none have yet been made. If
> we ever decide to make any, we should perhaps try to make them all at
> once and provide good conversion tools.
>
> Doug
>
> On Tue, Jul 16, 2013 at 12:40 AM, Christophe Taton <ta...@wibidata.com>
> wrote:
> > Hi Doug,
> >
> > Thanks for the explanation.
> > How far back is it reasonable to maintain backward and/or forward
> > compatibility?
> >
> > Assuming there is a limit to the forward compatibility requirement,
> could we
> > introduce the ability to read schemas with extended union descriptors,
> > without the ability to write such descriptors, and introduce the ability
> to
> > write after enough releases have passed?
> >
> > Thanks,
> > C.
> >
> >
> > On Mon, Jul 15, 2013 at 11:25 AM, Doug Cutting <cu...@apache.org>
> wrote:
> >>
> >> The reason is simply that the JSON syntax for union schemas doesn't
> >> permit properties. To support properties we'd need to add an
> >> alternate syntax for unions, but I don't see how to do that
> >> compatibly, so that an old client seeing one of the new union schemas
> >> would still be able to process the data.
> >>
> >> For example, the syntax might be something like:
> >>
> >> {"type":"union", "branches":[...], "prop1":"val1", ...}
> >>
> >> But that would cause errors in every existing implementation.
> >>
> >> Doug
> >>
> >> On Sat, Jul 13, 2013 at 7:01 PM, Christophe Taton <ta...@wibidata.com>
> >> wrote:
> >> > Hi,
> >> > I'm toying with a few changes to provide alternative representations
> of
> >> > union fields in Java (somewhat related to
> >> > https://issues.apache.org/jira/browse/AVRO-248).
> >> > To experiment with this, I'd like to set properties on union schemas,
> >> > but
> >> > properties are currently disabled on unions.
> >> > Is there a particular reason for this, or is it a reasonable change to
> >> > allow
> >> > properties on unions?
> >> > Thanks,
> >> > C.
> >
> >
>
Re: Properties on union schema?
Posted by Doug Cutting <cu...@apache.org>.
We've promised to maintain forward and backward compatibility until
version 2.0. There have been a number of incompatible schema changes
requested over the history of Avro, but none have yet been made. If
we ever decide to make any, we should perhaps try to make them all at
once and provide good conversion tools.
Doug
On Tue, Jul 16, 2013 at 12:40 AM, Christophe Taton <ta...@wibidata.com> wrote:
> Hi Doug,
>
> Thanks for the explanation.
> How far back is it reasonable to maintain backward and/or forward
> compatibility?
>
> Assuming there is a limit to the forward compatibility requirement, could we
> introduce the ability to read schemas with extended union descriptors,
> without the ability to write such descriptors, and introduce the ability to
> write after enough releases have passed?
>
> Thanks,
> C.
>
>
> On Mon, Jul 15, 2013 at 11:25 AM, Doug Cutting <cu...@apache.org> wrote:
>>
>> The reason is simply that the JSON syntax for union schemas doesn't
>> permit properties. To support properties we'd need to add an
>> alternate syntax for unions, but I don't see how to do that
>> compatibly, so that an old client seeing one of the new union schemas
>> would still be able to process the data.
>>
>> For example, the syntax might be something like:
>>
>> {"type":"union", "branches":[...], "prop1":"val1", ...}
>>
>> But that would cause errors in every existing implementation.
>>
>> Doug
>>
>> On Sat, Jul 13, 2013 at 7:01 PM, Christophe Taton <ta...@wibidata.com>
>> wrote:
>> > Hi,
>> > I'm toying with a few changes to provide alternative representations of
>> > union fields in Java (somewhat related to
>> > https://issues.apache.org/jira/browse/AVRO-248).
>> > To experiment with this, I'd like to set properties on union schemas,
>> > but
>> > properties are currently disabled on unions.
>> > Is there a particular reason for this, or is it a reasonable change to
>> > allow
>> > properties on unions?
>> > Thanks,
>> > C.
>
>
Re: Properties on union schema?
Posted by Christophe Taton <ta...@wibidata.com>.
Hi Doug,
Thanks for the explanation.
How far back is it reasonable to maintain backward and/or forward
compatibility?
Assuming there is a limit to the forward compatibility requirement, could
we introduce the ability to read schemas with extended union descriptors,
without the ability to write such descriptors, and introduce the ability to
write after enough releases have passed?
Thanks,
C.
On Mon, Jul 15, 2013 at 11:25 AM, Doug Cutting <cu...@apache.org> wrote:
> The reason is simply that the JSON syntax for union schemas doesn't
> permit properties. To support properties we'd need to add an
> alternate syntax for unions, but I don't see how to do that
> compatibly, so that an old client seeing one of the new union schemas
> would still be able to process the data.
>
> For example, the syntax might be something like:
>
> {"type":"union", "branches":[...], "prop1":"val1", ...}
>
> But that would cause errors in every existing implementation.
>
> Doug
>
> On Sat, Jul 13, 2013 at 7:01 PM, Christophe Taton <ta...@wibidata.com>
> wrote:
> > Hi,
> > I'm toying with a few changes to provide alternative representations of
> > union fields in Java (somewhat related to
> > https://issues.apache.org/jira/browse/AVRO-248).
> > To experiment with this, I'd like to set properties on union schemas, but
> > properties are currently disabled on unions.
> > Is there a particular reason for this, or is it a reasonable change to
> allow
> > properties on unions?
> > Thanks,
> > C.
>
Re: Properties on union schema?
Posted by Doug Cutting <cu...@apache.org>.
The reason is simply that the JSON syntax for union schemas doesn't
permit properties. To support properties we'd need to add an
alternate syntax for unions, but I don't see how to do that
compatibly, so that an old client seeing one of the new union schemas
would still be able to process the data.
For example, the syntax might be something like:
{"type":"union", "branches":[...], "prop1":"val1", ...}
But that would cause errors in every existing implementation.
Doug
On Sat, Jul 13, 2013 at 7:01 PM, Christophe Taton <ta...@wibidata.com> wrote:
> Hi,
> I'm toying with a few changes to provide alternative representations of
> union fields in Java (somewhat related to
> https://issues.apache.org/jira/browse/AVRO-248).
> To experiment with this, I'd like to set properties on union schemas, but
> properties are currently disabled on unions.
> Is there a particular reason for this, or is it a reasonable change to allow
> properties on unions?
> Thanks,
> C.