You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Lewis John Mcgibbney <le...@gmail.com> on 2013/11/08 21:19:54 UTC
Correct avsc definition for array of external object's
Hi Folks,
I am modeling the following Java as JSON in my avsc
/* list of {@link ExtractorSpec}s associated with this product type */
private List<ExtractorSpec> extractors = null;
...
{
"name": "extractors",
"type": {
"type": "array",
"items": "ExtractorSpec"
},
"doc": "A list of ExtractorSpecs associated with this product
type.",
"default": null
},
...
When I am compiling, as expected I am getting
./bin/gora goracompiler Product.avsc .
Exception in thread "main" org.apache.avro.SchemaParseException: Undefined
name: "ExtractorSpec".
The question I am asking is whether I should embed the ExtractorSpec as a
nested record? Or is there another way such as importing?
I've read a coule of threads [0] relating to IDL but this is not too
appealing as I *just* want my .avsc in JSON. I am trying to keeping things
as easy as possible for users.
I am using Avro 1.7.5 here.
Thank you very much in advance for any feedback.
Lewis
[0] http://www.mail-archive.com/user@avro.apache.org/msg00799.html
--
*Lewis*
Re: Correct avsc definition for array of external object's
Posted by Lewis John Mcgibbney <le...@gmail.com>.
I think now that I need to use imports to solve my problem. I'll look in to
that.
On Fri, Nov 8, 2013 at 8:19 PM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:
> Hi Folks,
> I am modeling the following Java as JSON in my avsc
>
> /* list of {@link ExtractorSpec}s associated with this product type */
> private List<ExtractorSpec> extractors = null;
>
> ...
> {
> "name": "extractors",
> "type": {
> "type": "array",
> "items": "ExtractorSpec"
> },
> "doc": "A list of ExtractorSpecs associated with this
> product type.",
> "default": null
> },
> ...
>
> When I am compiling, as expected I am getting
> ./bin/gora goracompiler Product.avsc .
> Exception in thread "main" org.apache.avro.SchemaParseException: Undefined
> name: "ExtractorSpec".
>
> The question I am asking is whether I should embed the ExtractorSpec as a
> nested record? Or is there another way such as importing?
> I've read a coule of threads [0] relating to IDL but this is not too
> appealing as I *just* want my .avsc in JSON. I am trying to keeping things
> as easy as possible for users.
> I am using Avro 1.7.5 here.
> Thank you very much in advance for any feedback.
> Lewis
>
> [0] http://www.mail-archive.com/user@avro.apache.org/msg00799.html
>
> --
> *Lewis*
>
--
*Lewis*
Re: Correct avsc definition for array of external object's
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi David,
Thanks for the heads up.
Lewis
On Fri, Nov 8, 2013 at 11:44 PM, David M. Carr <da...@carrclan.us> wrote:
> I'm currently working around the issue of inter-dependencies between Avro
> schema files in my Gradle Avro plugin. It uses a similar approach to what
> the command-line compiler does, making the previously parsed types
> available to later parsing operations. What it does differently is
> recognizing failures due to unrecognized types, and re-trying them later.
> This isn't the most efficient way to do it, but it allows compiling the
> schemas without manually specifying what order to process them.
>
> https://github.com/commercehub-oss/gradle-avro-plugin
>
>
> On Fri, Nov 8, 2013 at 3:48 PM, Lewis John Mcgibbney <
> lewis.mcgibbney@gmail.com> wrote:
>
>> Hi Doug,
>> Thanks for getting back to me.
>> Of course I read it all through, but its the last part of your comment
>> that has grabbed my attention.
>>
>> On Fri, Nov 8, 2013 at 8:36 PM, Doug Cutting <cu...@apache.org> wrote:
>>
>>>
>>> Nesting would certainly work, but may make things harder to maintain
>>> if you reference the type in several different .avsc files.
>>>
>>
>> +1. I realized this 5 mins after I sent the mail you replied to :)
>>
>>>
>>> But, from what I can tell, the Gora
>>> compiler does not currently support this (although it would not be
>>> hard to add).
>>>
>>
>> ... we've not needed it yet. :) However, yeah that doesn't mean it won't
>> happen int he future.
>>
>>
>>>
>>> Avro IDL does support imports, so you might use Avro's IDL compiler to
>>> convert .avdl files to .avsc files that expand nested types, then use
>>> the Gora compiler to compile these.
>>>
>>>
>> This sounds like just the ticket... AND you guys wrote the
>> avro-maven-plugin. How good is that ;)
>> Thanks for feedback.
>> Lewis
>>
>
>
>
> --
> David M. Carr
> david@carrclan.us
>
--
*Lewis*
Re: Correct avsc definition for array of external object's
Posted by "David M. Carr" <da...@carrclan.us>.
I'm currently working around the issue of inter-dependencies between Avro
schema files in my Gradle Avro plugin. It uses a similar approach to what
the command-line compiler does, making the previously parsed types
available to later parsing operations. What it does differently is
recognizing failures due to unrecognized types, and re-trying them later.
This isn't the most efficient way to do it, but it allows compiling the
schemas without manually specifying what order to process them.
https://github.com/commercehub-oss/gradle-avro-plugin
On Fri, Nov 8, 2013 at 3:48 PM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:
> Hi Doug,
> Thanks for getting back to me.
> Of course I read it all through, but its the last part of your comment
> that has grabbed my attention.
>
> On Fri, Nov 8, 2013 at 8:36 PM, Doug Cutting <cu...@apache.org> wrote:
>
>>
>> Nesting would certainly work, but may make things harder to maintain
>> if you reference the type in several different .avsc files.
>>
>
> +1. I realized this 5 mins after I sent the mail you replied to :)
>
>>
>> But, from what I can tell, the Gora
>> compiler does not currently support this (although it would not be
>> hard to add).
>>
>
> ... we've not needed it yet. :) However, yeah that doesn't mean it won't
> happen int he future.
>
>
>>
>> Avro IDL does support imports, so you might use Avro's IDL compiler to
>> convert .avdl files to .avsc files that expand nested types, then use
>> the Gora compiler to compile these.
>>
>>
> This sounds like just the ticket... AND you guys wrote the
> avro-maven-plugin. How good is that ;)
> Thanks for feedback.
> Lewis
>
--
David M. Carr
david@carrclan.us
Re: Correct avsc definition for array of external object's
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Doug,
Thanks for getting back to me.
Of course I read it all through, but its the last part of your comment that
has grabbed my attention.
On Fri, Nov 8, 2013 at 8:36 PM, Doug Cutting <cu...@apache.org> wrote:
>
> Nesting would certainly work, but may make things harder to maintain
> if you reference the type in several different .avsc files.
>
+1. I realized this 5 mins after I sent the mail you replied to :)
>
> But, from what I can tell, the Gora
> compiler does not currently support this (although it would not be
> hard to add).
>
... we've not needed it yet. :) However, yeah that doesn't mean it won't
happen int he future.
>
> Avro IDL does support imports, so you might use Avro's IDL compiler to
> convert .avdl files to .avsc files that expand nested types, then use
> the Gora compiler to compile these.
>
>
This sounds like just the ticket... AND you guys wrote the
avro-maven-plugin. How good is that ;)
Thanks for feedback.
Lewis
Re: Correct avsc definition for array of external object's
Posted by Doug Cutting <cu...@apache.org>.
On Fri, Nov 8, 2013 at 12:19 PM, Lewis John Mcgibbney
<le...@gmail.com> wrote:
> The question I am asking is whether I should embed the ExtractorSpec as a
> nested record? Or is there another way such as importing?
Nesting would certainly work, but may make things harder to maintain
if you reference the type in several different .avsc files.
Avro's schema parser and compiler supports pre-defining types. E.g.,
with Avro's command-line .avsc to .java compiler, types in files
defined earlier on the command line can be referenced by types in
files later on the command line. Similarly, the Java APIs and Maven
integrations support this. But, from what I can tell, the Gora
compiler does not currently support this (although it would not be
hard to add).
Avro IDL does support imports, so you might use Avro's IDL compiler to
convert .avdl files to .avsc files that expand nested types, then use
the Gora compiler to compile these.
Doug