You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by "Longton, Nigel" <Ni...@pimco.com> on 2014/03/13 15:34:08 UTC
Question about the state of the AVRO C# (csharp) implementation
Hi,
Are there plans to make C# feature equivalent? The major differences seem to be:
1. C# doesn't support the Avro IDL schema definition
2. C# doesn't support the JSON serialization codec
3. We had a case where we wanted to deserialize a json array which we can do in Java with Schema.createArray. The equivalent construct in C# was private and so not externally accessible.
We're making extensive use of Apache avro to communicate between Java and C# applications using JSON serialization rather than binary and in some cases we are creating schemas to match non-avro generated json from 'legacy' applications. I copied the Java json codec to C# and have got it working (I wont claim its fully tested though). More than happy to post this somewhere if its useful.
The other tweak we made was to change how 'default' values were serialized. We created a new writer classes that if the field value equaled the default it didn't serialize it, and a new reader class that set missing fields to the schema default if defined (missing non-defaulted fields are still an error). The advantage is that the JSON created is much cleaner and can be readily consumed by a jackson or any other json framework. (Currently these aren't ported to C# due to lack of time).
Are any of the above already scheduled to be added. Some of them might be contra to the avro approach though.
Thanks
Nigel
This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute, alter or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmissions cannot be guaranteed to be secure or without error as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender, therefore, does not accept liability for any errors or omissions in the contents of this message which arise during or as a result of e-mail transmission. If verification is required, please request a hard-copy version. This message is provided for information purposes and should not be construed as a solicitation or offer to buy or sell any securities or related financial instruments in any jurisdiction. Securities are offered in the U.S. through PIMCO Investments LLC, distributor and a company of PIMCO LLC.
RE: Question about the state of the AVRO C# (csharp) implementation
Posted by "Longton, Nigel" <Ni...@pimco.com>.
Doug,
Thanks for the prompt reply. I'll create a few jira issues and attach my code.
Cheers
Nigel
-----Original Message-----
From: Doug Cutting [mailto:cutting@apache.org]
Sent: Thursday, March 13, 2014 8:32 AM
To: user@avro.apache.org
Subject: Re: Question about the state of the AVRO C# (csharp) implementation
On Thu, Mar 13, 2014 at 7:34 AM, Longton, Nigel <Ni...@pimco.com> wrote:
> Are there plans to make C# feature equivalent?
As a volunteer-based open-source project, we don't have a long-term plan. Rather we, as a group, consider contributions as they arrive and generally accept those that aren't counter to other's needs.
>The major differences seem to be:
>
> 1. C# doesn’t support the Avro IDL schema definition
The IDL compiler does not need to be re-implemented per language. You can use the existing IDL compiler to convert IDL to protocols & schemas, then generate C# from those. A single implementation of the IDL compiler is easier to maintain.
> 2. C# doesn’t support the JSON serialization codec
>
> 3. We had a case where we wanted to deserialize a json array which we
> can do in Java with Schema.createArray. The equivalent construct in C#
> was private and so not externally accessible.
These would make great improvements to Avro C#. Please open issues in Jira.
> We’re making extensive use of Apache avro to communicate between Java
> and C# applications using JSON serialization rather than binary and in
> some cases we are creating schemas to match non-avro generated json from ‘legacy’
> applications. I copied the Java json codec to C# and have got it
> working (I wont claim its fully tested though). More than happy to
> post this somewhere if its useful.
This sounds useful. Please attach it as a patch in Jira.
https://cwiki.apache.org/confluence/display/AVRO/How+To+Contribute
> The other tweak we made was to change how ‘default’ values were serialized.
> We created a new writer classes that if the field value equaled the
> default it didn’t serialize it, and a new reader class that set
> missing fields to the schema default if defined (missing non-defaulted
> fields are still an error). The advantage is that the JSON created is
> much cleaner and can be readily consumed by a jackson or any other
> json framework. (Currently these aren’t ported to C# due to lack of time).
As you note, this is a non-standard mode of operation for Avro, but, if you find it useful, others might too. So long as it doesn't alter Avro's standard behavior we should consider adding it to Avro as an optional capability.
Cheers,
Doug
This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute, alter or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmissions cannot be guaranteed to be secure or without error as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender, therefore, does not accept liability for any errors or omissions in the contents of this message which arise during or as a result of e-mail transmission. If verification is required, please request a hard-copy version. This message is provided for information purposes and should not be construed as a solicitation or offer to buy or sell any securities or related financial instruments in any jurisdiction. Securities are offered in the U.S. through PIMCO Investments LLC, distributor and a company of PIMCO LLC.
Re: Question about the state of the AVRO C# (csharp) implementation
Posted by Doug Cutting <cu...@apache.org>.
On Thu, Mar 13, 2014 at 7:34 AM, Longton, Nigel <Ni...@pimco.com> wrote:
> Are there plans to make C# feature equivalent?
As a volunteer-based open-source project, we don't have a long-term
plan. Rather we, as a group, consider contributions as they arrive
and generally accept those that aren't counter to other's needs.
>The major differences seem to be:
>
> 1. C# doesn’t support the Avro IDL schema definition
The IDL compiler does not need to be re-implemented per language. You
can use the existing IDL compiler to convert IDL to protocols &
schemas, then generate C# from those. A single implementation of the
IDL compiler is easier to maintain.
> 2. C# doesn’t support the JSON serialization codec
>
> 3. We had a case where we wanted to deserialize a json array which we
> can do in Java with Schema.createArray. The equivalent construct in C# was
> private and so not externally accessible.
These would make great improvements to Avro C#. Please open issues in Jira.
> We’re making extensive use of Apache avro to communicate between Java and C#
> applications using JSON serialization rather than binary and in some cases
> we are creating schemas to match non-avro generated json from ‘legacy’
> applications. I copied the Java json codec to C# and have got it working (I
> wont claim its fully tested though). More than happy to post this somewhere
> if its useful.
This sounds useful. Please attach it as a patch in Jira.
https://cwiki.apache.org/confluence/display/AVRO/How+To+Contribute
> The other tweak we made was to change how ‘default’ values were serialized.
> We created a new writer classes that if the field value equaled the default
> it didn’t serialize it, and a new reader class that set missing fields to
> the schema default if defined (missing non-defaulted fields are still an
> error). The advantage is that the JSON created is much cleaner and can be
> readily consumed by a jackson or any other json framework. (Currently these
> aren’t ported to C# due to lack of time).
As you note, this is a non-standard mode of operation for Avro, but,
if you find it useful, others might too. So long as it doesn't alter
Avro's standard behavior we should consider adding it to Avro as an
optional capability.
Cheers,
Doug