Avro cycle support

[Bernhard Damberger <BD...@walmartlabs.com>]

My group is considering using Avro and we had some questions that I am hoping someone might have some insight into.

  1.  Is there a plan to add cycle support to Avro? I saw this ticket: https://issues.apache.org/jira/browse/AVRO-695. But it hasn't been worked on since 1/2011.
  2.  Why does Avro handle cyclical schema's but not cyclical objects?
  3.  Does anyone know when non-string map keys will be available (i.e. ticket https://issues.apache.org/jira/browse/AVRO-680)?
  4.  How does Avro handle inheritance in lists? For example: List<BaseClass> storing derived class objects?

Thanks for your help.

_bernhard

RE: Avro cycle support

["Pritchard, Charles X. -ND" <Ch...@disney.com>]

Vaguely; you're looking for a URI object.
Define it on your own terms and proxy your resolution objects through code.

Avro stops at schema definition: routing and transformations are left to the consumer.

As for direction on that part: I've tried to add maps... morphlines, elasticsearch index types, no success.
The URI construct simply means: take a string, execute code based on it, get some data.

Your easiest route in compatibility is to use UTF-8; your most efficient is to pass a byte array.


-Charles.



....................

________________________________________
From: Bernhard Damberger [BDamberger@walmartlabs.com]
Sent: Wednesday, May 14, 2014 10:38 AM
To: user@avro.apache.org
Subject: [SUSPECTED SPAM] Avro cycle support

My group is considering using Avro and we had some questions that I am hoping someone might have some insight into.

  1.  Is there a plan to add cycle support to Avro? I saw this ticket: https://issues.apache.org/jira/browse/AVRO-695. But it hasn't been worked on since 1/2011.
  2.  Why does Avro handle cyclical schema's but not cyclical objects?
  3.  Does anyone know when non-string map keys will be available (i.e. ticket https://issues.apache.org/jira/browse/AVRO-680)?
  4.  How does Avro handle inheritance in lists? For example: List<BaseClass> storing derived class objects?

Thanks for your help.

_bernhard

Re: Avro cycle support

[Doug Cutting <cu...@apache.org>]

On Wed, May 14, 2014 at 10:38 AM, Bernhard Damberger
<BD...@walmartlabs.com> wrote:
>   1.  Is there a plan to add cycle support to Avro? I saw this ticket: https://issues.apache.org/jira/browse/AVRO-695. But it hasn't been worked on since 1/2011.
>   2.  Why does Avro handle cyclical schema's but not cyclical objects?
>   3.  Does anyone know when non-string map keys will be available (i.e. ticket https://issues.apache.org/jira/browse/AVRO-680)?

In general, we don't centrally plan new Avro features.  Rather, if
someone contributes an implementation of a feature that everyone else
agrees does not conflict with their use of Avro then it is integrated.
 So the reason these have not yet been added is simply that no one has
provided an implementation that's compatible.  If these features are
critical to your use then you could either contribute a compatible
implementation yourself, or, if you work with a vendor, encourage your
vendor to contribute one.

>   4.  How does Avro handle inheritance in lists? For example: List<BaseClass> storing derived class objects?

Avro doesn't really handle inheritance.

You could use a union, explicitly adding new subclasses to the union
as they're implemented.  So if initially you have subclasses A and B
then the list element type could be the union [A,B].  Then when you
implement subclass C, update the union to be [A,B,C].

Alternately you might push the unions deeper.  The top-level record
would directly contain the fields of the "base class", and its fields
would have unions corresponding to the "subclass" variations.

Doug