You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@avro.apache.org by Elliot West <te...@gmail.com> on 2018/01/09 14:30:25 UTC

Schema composition; Circular dependencies across files

Hi,

I understand that Avro supports circular type dependencies, and also schema
composition. However, I cannot seem to be able to create a circular type
reference that spans multiple schema payloads. Consider this example:

*a.avsc*
{
  "type": "record", "name": "a", "fields": [{
    "name": "X", "type": "b"
  }]
}

*b.avsc*
{
  "type": "record", "name": "b", "fields": [{
    "name": "Y", "type": "c"
  }]
}

*c.avsc*
{
  "type": "record", "name": "c", "fields": [{
    "name": "Z", "type": "a"
  }]
}


In this case, there seems to be no good entry point that will result in a
valid initial schema, and hence while these types are declared in different
files, the total schema cannot be correctly parsed.

Can you confirm if this is an issue? Is there an outstanding JIRA, and if
not should I raise one. Can you suggest work arounds?

Thanks,

Elliot.

Re: Schema composition; Circular dependencies across files

Posted by Elliot West <te...@gmail.com>.

Thanks for the clarification Zoltan.

On 9 January 2018 at 18:18, Zoltan Farkas <zo...@yahoo.com> wrote:

> You will need to “bake in” everything in the avsc, so your a.avsc will
> look like:
>
> *a.avsc*
> {
>   "type": "record", "name": "a", "fields": [{
>     "name": "X", "type": {
>
>   "type": "record", "name": "b", "fields": [{
>     "name": "Y", "type": {
>
>   "type": "record", "name": "c", "fields": [{
>     "name": "Z", "type": "a"
>   }]
> }
>
>   }]
> }
>
>   }]
> }
>
>
> However this is not exactly easy to look at...
>
> I recommend using IDL to define/maintain schemas..., compile the IDL with
> the latest avro dev master, this will generate the schemas for you…
>
> cheers
>
> —Z
>
>
>
>
>
> On Jan 9, 2018, at 9:30 AM, Elliot West <te...@gmail.com> wrote:
>
> Hi,
>
> I understand that Avro supports circular type dependencies, and also
> schema composition. However, I cannot seem to be able to create a circular
> type reference that spans multiple schema payloads. Consider this example:
>
> *a.avsc*
> {
>   "type": "record", "name": "a", "fields": [{
>     "name": "X", "type": "b"
>   }]
> }
>
> *b.avsc*
> {
>   "type": "record", "name": "b", "fields": [{
>     "name": "Y", "type": "c"
>   }]
> }
>
> *c.avsc*
> {
>   "type": "record", "name": "c", "fields": [{
>     "name": "Z", "type": "a"
>   }]
> }
>
>
> In this case, there seems to be no good entry point that will result in a
> valid initial schema, and hence while these types are declared in different
> files, the total schema cannot be correctly parsed.
>
> Can you confirm if this is an issue? Is there an outstanding JIRA, and if
> not should I raise one. Can you suggest work arounds?
>
> Thanks,
>
> Elliot.
>
>
>

Re: Schema composition; Circular dependencies across files

Posted by Zoltan Farkas <zo...@yahoo.com>.

You will need to “bake in” everything in the avsc, so your a.avsc will look like:

> a.avsc
> {
>   "type": "record", "name": "a", "fields": [{
>     "name": "X", "type": {
>   "type": "record", "name": "b", "fields": [{
>     "name": "Y", "type": {
>   "type": "record", "name": "c", "fields": [{
>     "name": "Z", "type": "a"
>   }]
> }
>   }]
> }
>   }]
> }


However this is not exactly easy to look at...

I recommend using IDL to define/maintain schemas..., compile the IDL with the latest avro dev master, this will generate the schemas for you…

cheers

—Z





> On Jan 9, 2018, at 9:30 AM, Elliot West <te...@gmail.com> wrote:
> 
> Hi,
> 
> I understand that Avro supports circular type dependencies, and also schema composition. However, I cannot seem to be able to create a circular type reference that spans multiple schema payloads. Consider this example:
> 
> a.avsc
> {
>   "type": "record", "name": "a", "fields": [{
>     "name": "X", "type": "b"
>   }]
> }
> 
> b.avsc
> {
>   "type": "record", "name": "b", "fields": [{
>     "name": "Y", "type": "c"
>   }]
> }
> 
> c.avsc
> {
>   "type": "record", "name": "c", "fields": [{
>     "name": "Z", "type": "a"
>   }]
> }
> 
> In this case, there seems to be no good entry point that will result in a valid initial schema, and hence while these types are declared in different files, the total schema cannot be correctly parsed.
> 
> Can you confirm if this is an issue? Is there an outstanding JIRA, and if not should I raise one. Can you suggest work arounds?
> 
> Thanks,
> 
> Elliot.
>