You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by Mark Farnan <ma...@markfarnan.com> on 2022/12/12 17:18:33 UTC
Avro Schemas with external Dependancies - AVRO-3683
Hi All,
A week or so back, I created.
https://issues.apache.org/jira/browse/AVRO-3683 (AVRO RUST)
I've been trying to figure out if I can implement a solution, however my
rust skills are not up to the task as yet.
This is currently blocking my use of AVRO-RUST for a new project to
implement an Industry standard communications protocol, which is based on
AVRO. The standard makes extensive use of types referenced in other schema
files. There are some 200+ schema files in the standard.
On further investigation of the issue, I noticed that if a simple Schema
also has AvroSchema on it, then the schema generated from this WORKS for
the nested types, but the one generated by parse_list does not. (The
Schema from AvroSchema seems to resolve the nested types fine into the
resulting schema).
NOTE: I CANNOT use 'AvroSchema' for my industry standard schema's, as they
use features AvroSchema does not support. So need parse_list to create
valid schemas.
I think if parse_list was changed to create schema's which correctly
contained the referenced types, rather than by ref, then this would work
fine. I imagine it will also need logic to not duplicate any given schema
more than once.
Hoping someone can take a look at this.
Regards
Mark
Re: Avro Schemas with external Dependancies - AVRO-3683
Posted by Martin Grigorov <mg...@apache.org>.
Hi Mark,
On Mon, Dec 12, 2022 at 7:18 PM Mark Farnan <ma...@markfarnan.com> wrote:
> Hi All,
>
> A week or so back, I created.
> https://issues.apache.org/jira/browse/AVRO-3683 (AVRO RUST)
>
Yes, sorry! I have a deadline at work and I don't have much time for bigger
OSS tasks like this one right now.
> I've been trying to figure out if I can implement a solution, however my
> rust skills are not up to the task as yet.
>
> This is currently blocking my use of AVRO-RUST for a new project to
> implement an Industry standard communications protocol, which is based on
> AVRO. The standard makes extensive use of types referenced in other schema
> files. There are some 200+ schema files in the standard.
>
> On further investigation of the issue, I noticed that if a simple Schema
> also has AvroSchema on it, then the schema generated from this WORKS for
> the nested types, but the one generated by parse_list does not. (The
> Schema from AvroSchema seems to resolve the nested types fine into the
> resulting schema).
>
> NOTE: I CANNOT use 'AvroSchema' for my industry standard schema's, as they
> use features AvroSchema does not support. So need parse_list to create
> valid schemas.
>
> I think if parse_list was changed to create schema's which correctly
> contained the referenced types, rather than by ref, then this would work
> fine. I imagine it will also need logic to not duplicate any given schema
> more than once.
>
I think this is not the right approach here.
If it uses the referenced/expanded type then it won't be able to support
cyclic/self-referencing types, like the famous LinkedList example.
A better approach is to create ResolvedSchema from several Schemas, not
just one.
This way anything that currently accepts a single schema as parameter, e.g.
the Writer, could have the same factory method that accepts &[Schema]
I will try to implement it soon!
But if someone provides a PR in the meantime it would be awesome!
>
> Hoping someone can take a look at this.
>
> Regards
> Mark
>
Re: Avro Schemas with external Dependancies - AVRO-3683
Posted by Mark Farnan <ma...@markfarnan.com>.
Hi Jack.
I'll take a look at get_schema_in_ctx
AvroSchema and AvroSchemaComponent don't work for some of the types.
(Dosn't support FIXED for example), For reference, I'm using rsgen-avro
to make the rust types from the schemas. This mostly works, thought couple
minor tidy-ups needed).
Thing is, I have exact schema's for the messages, so trying to generate
them from the generated types, and dealing with new issues, seems somewhat
counter intuitive, when it would be better to use the provided schemas for
serialization.
One option I'm thinking of in the short term, is to pre-process the Schema
files to produce a single, fully resolved schema for each record I need to
serialize.
Given parse_list must already be resolving them, going to see if that can
help, (but my RUST is still pretty weak at this point). Part of the
purpose of this project, is to learn Rust and understand it's viability for
this sort of project.
Regards
Mark.
On Mon, Dec 12, 2022 at 6:31 PM Jack Klamer <jf...@gmail.com> wrote:
> Hey, can you investigate using ‘get_schema_in_ctxt’ for your use case?
> The AvroSchema derivation also implements the trait AvroSchemaComponent
> with allows it to handle the nested/recursive use case, but the raw method
> should be available to use, which should allow injecting schemas. The other
> thing I’ll mention is that the derive trait works for generic rust params.
> Aka you can have a type T that implements AvroSchemaComponent and use it
> within your structs. If your protocol needs to be genetic over multiple
> types you can create any number of container types with this param.
>
> Best,
> Jack
>
> On Mon, Dec 12, 2022 at 11:18 AM Mark Farnan <ma...@markfarnan.com> wrote:
>
> > Hi All,
> >
> > A week or so back, I created.
> > https://issues.apache.org/jira/browse/AVRO-3683 (AVRO RUST)
> >
> > I've been trying to figure out if I can implement a solution, however my
> > rust skills are not up to the task as yet.
> >
> > This is currently blocking my use of AVRO-RUST for a new project to
> > implement an Industry standard communications protocol, which is based on
> > AVRO. The standard makes extensive use of types referenced in other
> schema
> > files. There are some 200+ schema files in the standard.
> >
> > On further investigation of the issue, I noticed that if a simple Schema
> > also has AvroSchema on it, then the schema generated from this WORKS for
> > the nested types, but the one generated by parse_list does not. (The
> > Schema from AvroSchema seems to resolve the nested types fine into the
> > resulting schema).
> >
> > NOTE: I CANNOT use 'AvroSchema' for my industry standard schema's, as
> they
> > use features AvroSchema does not support. So need parse_list to create
> > valid schemas.
> >
> > I think if parse_list was changed to create schema's which correctly
> > contained the referenced types, rather than by ref, then this would work
> > fine. I imagine it will also need logic to not duplicate any given
> schema
> > more than once.
> >
> > Hoping someone can take a look at this.
> >
> > Regards
> > Mark
> >
>
Re: Avro Schemas with external Dependancies - AVRO-3683
Posted by Jack Klamer <jf...@gmail.com>.
Hey, can you investigate using ‘get_schema_in_ctxt’ for your use case?
The AvroSchema derivation also implements the trait AvroSchemaComponent
with allows it to handle the nested/recursive use case, but the raw method
should be available to use, which should allow injecting schemas. The other
thing I’ll mention is that the derive trait works for generic rust params.
Aka you can have a type T that implements AvroSchemaComponent and use it
within your structs. If your protocol needs to be genetic over multiple
types you can create any number of container types with this param.
Best,
Jack
On Mon, Dec 12, 2022 at 11:18 AM Mark Farnan <ma...@markfarnan.com> wrote:
> Hi All,
>
> A week or so back, I created.
> https://issues.apache.org/jira/browse/AVRO-3683 (AVRO RUST)
>
> I've been trying to figure out if I can implement a solution, however my
> rust skills are not up to the task as yet.
>
> This is currently blocking my use of AVRO-RUST for a new project to
> implement an Industry standard communications protocol, which is based on
> AVRO. The standard makes extensive use of types referenced in other schema
> files. There are some 200+ schema files in the standard.
>
> On further investigation of the issue, I noticed that if a simple Schema
> also has AvroSchema on it, then the schema generated from this WORKS for
> the nested types, but the one generated by parse_list does not. (The
> Schema from AvroSchema seems to resolve the nested types fine into the
> resulting schema).
>
> NOTE: I CANNOT use 'AvroSchema' for my industry standard schema's, as they
> use features AvroSchema does not support. So need parse_list to create
> valid schemas.
>
> I think if parse_list was changed to create schema's which correctly
> contained the referenced types, rather than by ref, then this would work
> fine. I imagine it will also need logic to not duplicate any given schema
> more than once.
>
> Hoping someone can take a look at this.
>
> Regards
> Mark
>