You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by Patrick Farry <pa...@gmail.com> on 2019/07/11 22:13:51 UTC
Re: C# POCO serializer/deserializer
Hi Brian,
Not meaning to hassle you (well maybe a little). I’ve got a few other enhancements we'd like to do as well as the Reflect code, and was hoping not to have to maintain a bunch of branches.
Here are the planned changes:
- update newtonsoft.json and add the json path whenever the schema parser reports an error - we have a number of large schemas and debugging them is proving difficult
- Roslyn based schema generator from C# class
- Reflect option added to codegen (possibly updating codegen to use Roslyn)
- Serialize to json
Thanks.
> On May 2, 2019, at 6:22 PM, Brian Lachniet <bl...@gmail.com> wrote:
>
> Hey Patrick,
>
> This sounds very useful! I'd love to see this introduced to the C# library.
>
> On Thu, May 2, 2019 at 5:47 PM Ivan Greene <ig...@fanthreesixty.com>
> wrote:
>
>> Patrick,
>>
>> This sounds as though it would be the C# equivalent of Java's ReflectData,
>> which has been part of the Avro API for several years now, so it’s likely
>> there would be interest. Your best bet is to open a Jira describing the
>> planned feature and open a pull request on Github to start the discussion.
>> The contributing page on the wiki is a bit out of date and still recommends
>> submitting patches on Jira but I believe that Github is now the official
>> repository.
>>
>>> On May 1, 2019, at 4:31 PM, Patrick Farry <pa...@gmail.com>
>> wrote:
>>>
>>> Hi All,
>>>
>>> I have written code that implements Avro serialization for POCO classes
>> - i.e. classes that are not generated by Avro codegen and do not implement
>> ISpecificRecord. The idea was to make it work as much like JSON.net as
>> possible.
>>>
>>> The serializer inherits from SpecificDefaultWriter and the deserializer
>> from SpecificDefaultReader.
>>>
>>> Avro fields are mapped to C# properties either by matching the field
>> name and property name or by using an attribute to specify the field
>> sequence number.
>>>
>>> Is this something that would be of interest to the Avro project? My
>> company has approved committing the code and I’d be available and happy to
>> maintain this code and work on other parts of the C# implementation.
>>>
>>> Regards.
>>>
>>>
>>
>>
>
> --
>
> [image: 51b630b05e01a6d5134ccfd520f547c4.png]
>
> Brian Lachniet
>
> Software Engineer
>
> E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
>
> <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
Re: C# POCO serializer/deserializer
Posted by Patrick Farry <pa...@gmail.com>.
Apologies but I had to make everything StyleCop compliant for work. Everything is basically the same but there are lot of spaces added/removed and a couple of new files because it doesn’t like multiple classes in a file. Makes it a pain to review.
> On Jul 11, 2019, at 5:05 PM, Brian Lachniet <bl...@gmail.com> wrote:
>
> Hi Patrick!
>
> I'm sorry I haven't gotten back to the Reflect review yet. I'm hoping to
> get back on that this weekend.
>
> That sounds like a good list of additions. I'm particularly interested in
> having support for JSON encoding/decoding in C#. I bet we could get a long
> ways simply porting the JsonEncoder
> <https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/io/JsonEncoder.java>
> from the Java bindings.
>
> I agree that we need to bump the Newtonsoft.Json verison. That just came up
> in another issue recently. What version of Newtonsoft.Json do you think we
> should upgrade to? I'm a little wary of bumping all the way to the latest,
> as that would force everyone using Avro to also go to the latest version.
> Maybe that's not a bad thing, I'm not sure.
>
> I'm not familiar with Roslyn-based code generation. I'll do some Googling
> on that (if you've got some good articles or starting points to share, I'd
> appreciate that). Codegen currently uses the System.CodeDom API. Would
> Roslyn-based code generation be a replacement for that, or am I
> misunderstanding something?
>
> Thank you for your patience so far, Patrick. I'm looking forward to these
> new features too!
>
> On Thu, Jul 11, 2019 at 6:14 PM Patrick Farry <pa...@gmail.com>
> wrote:
>
>> Hi Brian,
>>
>> Not meaning to hassle you (well maybe a little). I’ve got a few other
>> enhancements we'd like to do as well as the Reflect code, and was hoping
>> not to have to maintain a bunch of branches.
>>
>> Here are the planned changes:
>> - update newtonsoft.json and add the json path whenever the schema parser
>> reports an error - we have a number of large schemas and debugging them is
>> proving difficult
>> - Roslyn based schema generator from C# class
>> - Reflect option added to codegen (possibly updating codegen to use Roslyn)
>> - Serialize to json
>>
>> Thanks.
>>
>>> On May 2, 2019, at 6:22 PM, Brian Lachniet <bl...@gmail.com> wrote:
>>>
>>> Hey Patrick,
>>>
>>> This sounds very useful! I'd love to see this introduced to the C#
>> library.
>>>
>>> On Thu, May 2, 2019 at 5:47 PM Ivan Greene <ig...@fanthreesixty.com>
>>> wrote:
>>>
>>>> Patrick,
>>>>
>>>> This sounds as though it would be the C# equivalent of Java's
>> ReflectData,
>>>> which has been part of the Avro API for several years now, so it’s
>> likely
>>>> there would be interest. Your best bet is to open a Jira describing the
>>>> planned feature and open a pull request on Github to start the
>> discussion.
>>>> The contributing page on the wiki is a bit out of date and still
>> recommends
>>>> submitting patches on Jira but I believe that Github is now the official
>>>> repository.
>>>>
>>>>> On May 1, 2019, at 4:31 PM, Patrick Farry <pa...@gmail.com>
>>>> wrote:
>>>>>
>>>>> Hi All,
>>>>>
>>>>> I have written code that implements Avro serialization for POCO classes
>>>> - i.e. classes that are not generated by Avro codegen and do not
>> implement
>>>> ISpecificRecord. The idea was to make it work as much like JSON.net as
>>>> possible.
>>>>>
>>>>> The serializer inherits from SpecificDefaultWriter and the deserializer
>>>> from SpecificDefaultReader.
>>>>>
>>>>> Avro fields are mapped to C# properties either by matching the field
>>>> name and property name or by using an attribute to specify the field
>>>> sequence number.
>>>>>
>>>>> Is this something that would be of interest to the Avro project? My
>>>> company has approved committing the code and I’d be available and happy
>> to
>>>> maintain this code and work on other parts of the C# implementation.
>>>>>
>>>>> Regards.
>>>>>
>>>>>
>>>>
>>>>
>>>
>>> --
>>>
>>> [image: 51b630b05e01a6d5134ccfd520f547c4.png]
>>>
>>> Brian Lachniet
>>>
>>> Software Engineer
>>>
>>> E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
>>>
>>> <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
>>
>>
>
> --
>
> [image: 51b630b05e01a6d5134ccfd520f547c4.png]
>
> Brian Lachniet
>
> Software Engineer
>
> E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
>
> <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
Re: C# POCO serializer/deserializer
Posted by Patrick Farry <pa...@gmail.com>.
Hi Brian,
Thanks for the reply (and for shepherding the project).
Regarding Json. I’ll take a look at the Java code. I’ve also used Newtonsoft for things like this previously. JsonTextReader/JsonTextWriter look like they would map easily to the Avro Encode/Decoder interfaces. The banner on the Json.NET <http://json.net/> documentation says "JsonTextWriter Class Represents a writer that provides a fast, non-cached, forward-only way of generating JSON data.”, which seems like what we need. https://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_JsonTextWriter.htm <https://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_JsonTextWriter.htm>
Regarding Newtonsoft version. Maybe 10.0.3? It is 2 years old and 2 major versions back. It is also by far the most popular download on nuget. For what I’m trying to do (better error reporting while parsing) I just need the JToken to to have the Path property.
Regarding Roslyn. This is a detailed intro/tutorial https://medium.com/@CPP_Coder/introduction-to-roslyn-and-its-use-in-program-development-bce2043fc45d <https://medium.com/@CPP_Coder/introduction-to-roslyn-and-its-use-in-program-development-bce2043fc45d>. It is the compiler framework that’s being used for code analysis tools, cross compilers etc. For code->schema I’m thinking to do the following:
- open the solution
- use the syntax walker https://joshvarty.com/2014/07/26/learn-roslyn-now-part-4-csharpsyntaxwalker/ <https://joshvarty.com/2014/07/26/learn-roslyn-now-part-4-csharpsyntaxwalker/> to parse the required class, then generate a schema (or possibly a protocol for an entire namespace).
I haven’t thought a lot about the code generation side. But what I’m hoping is that we can parse the C#, as with schema generation, and then generate updates to the code by modifying the syntax tree. This will have the effect of editing the C# and preserving any edits, unmapped properties, and comments the developer has made. https://msdn.microsoft.com/en-us/magazine/mt808499.aspx <https://msdn.microsoft.com/en-us/magazine/mt808499.aspx>. As I’m typing this I think the reverse for the schema would be good too - load C# and schema and then edit the schema tree to sync any changes rather than just regenerating it every time.
> On Jul 11, 2019, at 5:05 PM, Brian Lachniet <bl...@gmail.com> wrote:
>
> Hi Patrick!
>
> I'm sorry I haven't gotten back to the Reflect review yet. I'm hoping to
> get back on that this weekend.
>
> That sounds like a good list of additions. I'm particularly interested in
> having support for JSON encoding/decoding in C#. I bet we could get a long
> ways simply porting the JsonEncoder
> <https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/io/JsonEncoder.java>
> from the Java bindings.
>
> I agree that we need to bump the Newtonsoft.Json verison. That just came up
> in another issue recently. What version of Newtonsoft.Json do you think we
> should upgrade to? I'm a little wary of bumping all the way to the latest,
> as that would force everyone using Avro to also go to the latest version.
> Maybe that's not a bad thing, I'm not sure.
>
> I'm not familiar with Roslyn-based code generation. I'll do some Googling
> on that (if you've got some good articles or starting points to share, I'd
> appreciate that). Codegen currently uses the System.CodeDom API. Would
> Roslyn-based code generation be a replacement for that, or am I
> misunderstanding something?
>
> Thank you for your patience so far, Patrick. I'm looking forward to these
> new features too!
>
> On Thu, Jul 11, 2019 at 6:14 PM Patrick Farry <pa...@gmail.com>
> wrote:
>
>> Hi Brian,
>>
>> Not meaning to hassle you (well maybe a little). I’ve got a few other
>> enhancements we'd like to do as well as the Reflect code, and was hoping
>> not to have to maintain a bunch of branches.
>>
>> Here are the planned changes:
>> - update newtonsoft.json and add the json path whenever the schema parser
>> reports an error - we have a number of large schemas and debugging them is
>> proving difficult
>> - Roslyn based schema generator from C# class
>> - Reflect option added to codegen (possibly updating codegen to use Roslyn)
>> - Serialize to json
>>
>> Thanks.
>>
>>> On May 2, 2019, at 6:22 PM, Brian Lachniet <bl...@gmail.com> wrote:
>>>
>>> Hey Patrick,
>>>
>>> This sounds very useful! I'd love to see this introduced to the C#
>> library.
>>>
>>> On Thu, May 2, 2019 at 5:47 PM Ivan Greene <ig...@fanthreesixty.com>
>>> wrote:
>>>
>>>> Patrick,
>>>>
>>>> This sounds as though it would be the C# equivalent of Java's
>> ReflectData,
>>>> which has been part of the Avro API for several years now, so it’s
>> likely
>>>> there would be interest. Your best bet is to open a Jira describing the
>>>> planned feature and open a pull request on Github to start the
>> discussion.
>>>> The contributing page on the wiki is a bit out of date and still
>> recommends
>>>> submitting patches on Jira but I believe that Github is now the official
>>>> repository.
>>>>
>>>>> On May 1, 2019, at 4:31 PM, Patrick Farry <pa...@gmail.com>
>>>> wrote:
>>>>>
>>>>> Hi All,
>>>>>
>>>>> I have written code that implements Avro serialization for POCO classes
>>>> - i.e. classes that are not generated by Avro codegen and do not
>> implement
>>>> ISpecificRecord. The idea was to make it work as much like JSON.net as
>>>> possible.
>>>>>
>>>>> The serializer inherits from SpecificDefaultWriter and the deserializer
>>>> from SpecificDefaultReader.
>>>>>
>>>>> Avro fields are mapped to C# properties either by matching the field
>>>> name and property name or by using an attribute to specify the field
>>>> sequence number.
>>>>>
>>>>> Is this something that would be of interest to the Avro project? My
>>>> company has approved committing the code and I’d be available and happy
>> to
>>>> maintain this code and work on other parts of the C# implementation.
>>>>>
>>>>> Regards.
>>>>>
>>>>>
>>>>
>>>>
>>>
>>> --
>>>
>>> [image: 51b630b05e01a6d5134ccfd520f547c4.png]
>>>
>>> Brian Lachniet
>>>
>>> Software Engineer
>>>
>>> E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
>>>
>>> <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
>>
>>
>
> --
>
> [image: 51b630b05e01a6d5134ccfd520f547c4.png]
>
> Brian Lachniet
>
> Software Engineer
>
> E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
>
> <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
Re: C# POCO serializer/deserializer
Posted by Patrick Farry <pa...@gmail.com>.
The schema generator is checked in as a separate project for now. It takes a number of C# classes and generates an Avro protocol using the Microsoft.CodeAnalysis package.
https://github.com/pa009fa/schemagen <https://github.com/pa009fa/schemagen>
There is a driver program which runs with the following parameters:
$ dotnet run -- --help
schemagen 1.0.0
Copyright (C) 2019 schemagen
-i, --include Required. Includes
-o, --outfile (Default: .) Output file. . for console
-t, --target Required. File name of the type to generate
-d, --defaultconverters Required. Default converters
-c, --converters Required. Default converters
--help Display this help screen.
--version Display version information.
The VSCode debugger is set up to generate a protocol from the classes in the csharp folder.
There are a number of new attributes to help when specifying the schema. Examples are in the Z.cs file. Attribute specify unions, default values etc.
The Reflect array helpers are not supported yet.
Any and all feedback is welcome.
At this stage I’m not sure it’s ready fro a detailed code review (but that would be fine as well if anyone has the time).
Also, any thoughts on how (whether?) to integrate it into the project are also welcome.
Cheers.
> On Aug 10, 2019, at 10:36 AM, Brian Lachniet <bl...@gmail.com> wrote:
>
> Hey Patrick,
>
> That sounds pretty cool. I think that the best way to handle this right
> now, if you believe there is additional work to make it ready for more
> general use, is to keep it on a separate branch in your repository.
>
> I wonder if this sort of functionality exists in other language bindings
> (I'm really only familiar with the C# bindings). For those familiar with
> the other languages, has this kind of thing been done before?
>
> I agree that the C# bindings could really benefit from having a JSON
> encoder. And yes, we should definitely take advantage of the JSON.net
> library. In implementation, I imagine this would take the form of new
> implementations of the *Encoder* and *Decoder* interfaces, *JsonEncoder *and
> *JsonDecoder*.
>
> On Mon, Aug 5, 2019 at 11:56 PM Patrick Farry <pa...@gmail.com>
> wrote:
>
>> Hi Brian,
>>
>> I got the schema from C# source code generator working, at least well
>> enough for us. Not sure it is ready for general use but if you’re
>> interested I can send the code. What is the usual practice for things under
>> development - should I create a PR even if the code is some way off from
>> being mergeable or just a branch of my repo?
>>
>> Our use case is that the developers write C# DTO classes and don’t need to
>> worry much about the schema. We build a protocol from the classes and then
>> schemas for each top level object from the protocol.
>>
>> Also, I think we (Pitney Bowes) need to get something going about a JSON
>> encoder. Shifting to Avro is a bit of a stretch for some of our developers
>> since they can’t easily see the message contents without additional
>> decoding tools. Are you ok with basing it on JSON.net? It should be pretty
>> fast to get something working.
>>
>>> On Jul 11, 2019, at 5:05 PM, Brian Lachniet <bl...@gmail.com> wrote:
>>>
>>> Hi Patrick!
>>>
>>> I'm sorry I haven't gotten back to the Reflect review yet. I'm hoping to
>>> get back on that this weekend.
>>>
>>> That sounds like a good list of additions. I'm particularly interested in
>>> having support for JSON encoding/decoding in C#. I bet we could get a
>> long
>>> ways simply porting the JsonEncoder
>>> <
>> https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/io/JsonEncoder.java
>>>
>>> from the Java bindings.
>>>
>>> I agree that we need to bump the Newtonsoft.Json verison. That just came
>> up
>>> in another issue recently. What version of Newtonsoft.Json do you think
>> we
>>> should upgrade to? I'm a little wary of bumping all the way to the
>> latest,
>>> as that would force everyone using Avro to also go to the latest version.
>>> Maybe that's not a bad thing, I'm not sure.
>>>
>>> I'm not familiar with Roslyn-based code generation. I'll do some Googling
>>> on that (if you've got some good articles or starting points to share,
>> I'd
>>> appreciate that). Codegen currently uses the System.CodeDom API. Would
>>> Roslyn-based code generation be a replacement for that, or am I
>>> misunderstanding something?
>>>
>>> Thank you for your patience so far, Patrick. I'm looking forward to these
>>> new features too!
>>>
>>> On Thu, Jul 11, 2019 at 6:14 PM Patrick Farry <patrick.s.farry@gmail.com
>>>
>>> wrote:
>>>
>>>> Hi Brian,
>>>>
>>>> Not meaning to hassle you (well maybe a little). I’ve got a few other
>>>> enhancements we'd like to do as well as the Reflect code, and was hoping
>>>> not to have to maintain a bunch of branches.
>>>>
>>>> Here are the planned changes:
>>>> - update newtonsoft.json and add the json path whenever the schema
>> parser
>>>> reports an error - we have a number of large schemas and debugging them
>> is
>>>> proving difficult
>>>> - Roslyn based schema generator from C# class
>>>> - Reflect option added to codegen (possibly updating codegen to use
>> Roslyn)
>>>> - Serialize to json
>>>>
>>>> Thanks.
>>>>
>>>>> On May 2, 2019, at 6:22 PM, Brian Lachniet <bl...@gmail.com>
>> wrote:
>>>>>
>>>>> Hey Patrick,
>>>>>
>>>>> This sounds very useful! I'd love to see this introduced to the C#
>>>> library.
>>>>>
>>>>> On Thu, May 2, 2019 at 5:47 PM Ivan Greene <ig...@fanthreesixty.com>
>>>>> wrote:
>>>>>
>>>>>> Patrick,
>>>>>>
>>>>>> This sounds as though it would be the C# equivalent of Java's
>>>> ReflectData,
>>>>>> which has been part of the Avro API for several years now, so it’s
>>>> likely
>>>>>> there would be interest. Your best bet is to open a Jira describing
>> the
>>>>>> planned feature and open a pull request on Github to start the
>>>> discussion.
>>>>>> The contributing page on the wiki is a bit out of date and still
>>>> recommends
>>>>>> submitting patches on Jira but I believe that Github is now the
>> official
>>>>>> repository.
>>>>>>
>>>>>>> On May 1, 2019, at 4:31 PM, Patrick Farry <patrick.s.farry@gmail.com
>>>
>>>>>> wrote:
>>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> I have written code that implements Avro serialization for POCO
>> classes
>>>>>> - i.e. classes that are not generated by Avro codegen and do not
>>>> implement
>>>>>> ISpecificRecord. The idea was to make it work as much like JSON.net as
>>>>>> possible.
>>>>>>>
>>>>>>> The serializer inherits from SpecificDefaultWriter and the
>> deserializer
>>>>>> from SpecificDefaultReader.
>>>>>>>
>>>>>>> Avro fields are mapped to C# properties either by matching the field
>>>>>> name and property name or by using an attribute to specify the field
>>>>>> sequence number.
>>>>>>>
>>>>>>> Is this something that would be of interest to the Avro project? My
>>>>>> company has approved committing the code and I’d be available and
>> happy
>>>> to
>>>>>> maintain this code and work on other parts of the C# implementation.
>>>>>>>
>>>>>>> Regards.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> [image: 51b630b05e01a6d5134ccfd520f547c4.png]
>>>>>
>>>>> Brian Lachniet
>>>>>
>>>>> Software Engineer
>>>>>
>>>>> E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
>>>>>
>>>>> <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
>>>>
>>>>
>>>
>>> --
>>>
>>> [image: 51b630b05e01a6d5134ccfd520f547c4.png]
>>>
>>> Brian Lachniet
>>>
>>> Software Engineer
>>>
>>> E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
>>>
>>> <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
>>
>>
>
> --
>
> [image: 51b630b05e01a6d5134ccfd520f547c4.png]
>
> Brian Lachniet
>
> Software Engineer
>
> E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
>
> <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
Re: C# POCO serializer/deserializer
Posted by Brian Lachniet <bl...@gmail.com>.
Hey Patrick,
That sounds pretty cool. I think that the best way to handle this right
now, if you believe there is additional work to make it ready for more
general use, is to keep it on a separate branch in your repository.
I wonder if this sort of functionality exists in other language bindings
(I'm really only familiar with the C# bindings). For those familiar with
the other languages, has this kind of thing been done before?
I agree that the C# bindings could really benefit from having a JSON
encoder. And yes, we should definitely take advantage of the JSON.net
library. In implementation, I imagine this would take the form of new
implementations of the *Encoder* and *Decoder* interfaces, *JsonEncoder *and
*JsonDecoder*.
On Mon, Aug 5, 2019 at 11:56 PM Patrick Farry <pa...@gmail.com>
wrote:
> Hi Brian,
>
> I got the schema from C# source code generator working, at least well
> enough for us. Not sure it is ready for general use but if you’re
> interested I can send the code. What is the usual practice for things under
> development - should I create a PR even if the code is some way off from
> being mergeable or just a branch of my repo?
>
> Our use case is that the developers write C# DTO classes and don’t need to
> worry much about the schema. We build a protocol from the classes and then
> schemas for each top level object from the protocol.
>
> Also, I think we (Pitney Bowes) need to get something going about a JSON
> encoder. Shifting to Avro is a bit of a stretch for some of our developers
> since they can’t easily see the message contents without additional
> decoding tools. Are you ok with basing it on JSON.net? It should be pretty
> fast to get something working.
>
> > On Jul 11, 2019, at 5:05 PM, Brian Lachniet <bl...@gmail.com> wrote:
> >
> > Hi Patrick!
> >
> > I'm sorry I haven't gotten back to the Reflect review yet. I'm hoping to
> > get back on that this weekend.
> >
> > That sounds like a good list of additions. I'm particularly interested in
> > having support for JSON encoding/decoding in C#. I bet we could get a
> long
> > ways simply porting the JsonEncoder
> > <
> https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/io/JsonEncoder.java
> >
> > from the Java bindings.
> >
> > I agree that we need to bump the Newtonsoft.Json verison. That just came
> up
> > in another issue recently. What version of Newtonsoft.Json do you think
> we
> > should upgrade to? I'm a little wary of bumping all the way to the
> latest,
> > as that would force everyone using Avro to also go to the latest version.
> > Maybe that's not a bad thing, I'm not sure.
> >
> > I'm not familiar with Roslyn-based code generation. I'll do some Googling
> > on that (if you've got some good articles or starting points to share,
> I'd
> > appreciate that). Codegen currently uses the System.CodeDom API. Would
> > Roslyn-based code generation be a replacement for that, or am I
> > misunderstanding something?
> >
> > Thank you for your patience so far, Patrick. I'm looking forward to these
> > new features too!
> >
> > On Thu, Jul 11, 2019 at 6:14 PM Patrick Farry <patrick.s.farry@gmail.com
> >
> > wrote:
> >
> >> Hi Brian,
> >>
> >> Not meaning to hassle you (well maybe a little). I’ve got a few other
> >> enhancements we'd like to do as well as the Reflect code, and was hoping
> >> not to have to maintain a bunch of branches.
> >>
> >> Here are the planned changes:
> >> - update newtonsoft.json and add the json path whenever the schema
> parser
> >> reports an error - we have a number of large schemas and debugging them
> is
> >> proving difficult
> >> - Roslyn based schema generator from C# class
> >> - Reflect option added to codegen (possibly updating codegen to use
> Roslyn)
> >> - Serialize to json
> >>
> >> Thanks.
> >>
> >>> On May 2, 2019, at 6:22 PM, Brian Lachniet <bl...@gmail.com>
> wrote:
> >>>
> >>> Hey Patrick,
> >>>
> >>> This sounds very useful! I'd love to see this introduced to the C#
> >> library.
> >>>
> >>> On Thu, May 2, 2019 at 5:47 PM Ivan Greene <ig...@fanthreesixty.com>
> >>> wrote:
> >>>
> >>>> Patrick,
> >>>>
> >>>> This sounds as though it would be the C# equivalent of Java's
> >> ReflectData,
> >>>> which has been part of the Avro API for several years now, so it’s
> >> likely
> >>>> there would be interest. Your best bet is to open a Jira describing
> the
> >>>> planned feature and open a pull request on Github to start the
> >> discussion.
> >>>> The contributing page on the wiki is a bit out of date and still
> >> recommends
> >>>> submitting patches on Jira but I believe that Github is now the
> official
> >>>> repository.
> >>>>
> >>>>> On May 1, 2019, at 4:31 PM, Patrick Farry <patrick.s.farry@gmail.com
> >
> >>>> wrote:
> >>>>>
> >>>>> Hi All,
> >>>>>
> >>>>> I have written code that implements Avro serialization for POCO
> classes
> >>>> - i.e. classes that are not generated by Avro codegen and do not
> >> implement
> >>>> ISpecificRecord. The idea was to make it work as much like JSON.net as
> >>>> possible.
> >>>>>
> >>>>> The serializer inherits from SpecificDefaultWriter and the
> deserializer
> >>>> from SpecificDefaultReader.
> >>>>>
> >>>>> Avro fields are mapped to C# properties either by matching the field
> >>>> name and property name or by using an attribute to specify the field
> >>>> sequence number.
> >>>>>
> >>>>> Is this something that would be of interest to the Avro project? My
> >>>> company has approved committing the code and I’d be available and
> happy
> >> to
> >>>> maintain this code and work on other parts of the C# implementation.
> >>>>>
> >>>>> Regards.
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>> --
> >>>
> >>> [image: 51b630b05e01a6d5134ccfd520f547c4.png]
> >>>
> >>> Brian Lachniet
> >>>
> >>> Software Engineer
> >>>
> >>> E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
> >>>
> >>> <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
> >>
> >>
> >
> > --
> >
> > [image: 51b630b05e01a6d5134ccfd520f547c4.png]
> >
> > Brian Lachniet
> >
> > Software Engineer
> >
> > E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
> >
> > <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
>
>
--
[image: 51b630b05e01a6d5134ccfd520f547c4.png]
Brian Lachniet
Software Engineer
E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
<https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
Re: C# POCO serializer/deserializer
Posted by Patrick Farry <pa...@gmail.com>.
Hi Brian,
I got the schema from C# source code generator working, at least well enough for us. Not sure it is ready for general use but if you’re interested I can send the code. What is the usual practice for things under development - should I create a PR even if the code is some way off from being mergeable or just a branch of my repo?
Our use case is that the developers write C# DTO classes and don’t need to worry much about the schema. We build a protocol from the classes and then schemas for each top level object from the protocol.
Also, I think we (Pitney Bowes) need to get something going about a JSON encoder. Shifting to Avro is a bit of a stretch for some of our developers since they can’t easily see the message contents without additional decoding tools. Are you ok with basing it on JSON.net? It should be pretty fast to get something working.
> On Jul 11, 2019, at 5:05 PM, Brian Lachniet <bl...@gmail.com> wrote:
>
> Hi Patrick!
>
> I'm sorry I haven't gotten back to the Reflect review yet. I'm hoping to
> get back on that this weekend.
>
> That sounds like a good list of additions. I'm particularly interested in
> having support for JSON encoding/decoding in C#. I bet we could get a long
> ways simply porting the JsonEncoder
> <https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/io/JsonEncoder.java>
> from the Java bindings.
>
> I agree that we need to bump the Newtonsoft.Json verison. That just came up
> in another issue recently. What version of Newtonsoft.Json do you think we
> should upgrade to? I'm a little wary of bumping all the way to the latest,
> as that would force everyone using Avro to also go to the latest version.
> Maybe that's not a bad thing, I'm not sure.
>
> I'm not familiar with Roslyn-based code generation. I'll do some Googling
> on that (if you've got some good articles or starting points to share, I'd
> appreciate that). Codegen currently uses the System.CodeDom API. Would
> Roslyn-based code generation be a replacement for that, or am I
> misunderstanding something?
>
> Thank you for your patience so far, Patrick. I'm looking forward to these
> new features too!
>
> On Thu, Jul 11, 2019 at 6:14 PM Patrick Farry <pa...@gmail.com>
> wrote:
>
>> Hi Brian,
>>
>> Not meaning to hassle you (well maybe a little). I’ve got a few other
>> enhancements we'd like to do as well as the Reflect code, and was hoping
>> not to have to maintain a bunch of branches.
>>
>> Here are the planned changes:
>> - update newtonsoft.json and add the json path whenever the schema parser
>> reports an error - we have a number of large schemas and debugging them is
>> proving difficult
>> - Roslyn based schema generator from C# class
>> - Reflect option added to codegen (possibly updating codegen to use Roslyn)
>> - Serialize to json
>>
>> Thanks.
>>
>>> On May 2, 2019, at 6:22 PM, Brian Lachniet <bl...@gmail.com> wrote:
>>>
>>> Hey Patrick,
>>>
>>> This sounds very useful! I'd love to see this introduced to the C#
>> library.
>>>
>>> On Thu, May 2, 2019 at 5:47 PM Ivan Greene <ig...@fanthreesixty.com>
>>> wrote:
>>>
>>>> Patrick,
>>>>
>>>> This sounds as though it would be the C# equivalent of Java's
>> ReflectData,
>>>> which has been part of the Avro API for several years now, so it’s
>> likely
>>>> there would be interest. Your best bet is to open a Jira describing the
>>>> planned feature and open a pull request on Github to start the
>> discussion.
>>>> The contributing page on the wiki is a bit out of date and still
>> recommends
>>>> submitting patches on Jira but I believe that Github is now the official
>>>> repository.
>>>>
>>>>> On May 1, 2019, at 4:31 PM, Patrick Farry <pa...@gmail.com>
>>>> wrote:
>>>>>
>>>>> Hi All,
>>>>>
>>>>> I have written code that implements Avro serialization for POCO classes
>>>> - i.e. classes that are not generated by Avro codegen and do not
>> implement
>>>> ISpecificRecord. The idea was to make it work as much like JSON.net as
>>>> possible.
>>>>>
>>>>> The serializer inherits from SpecificDefaultWriter and the deserializer
>>>> from SpecificDefaultReader.
>>>>>
>>>>> Avro fields are mapped to C# properties either by matching the field
>>>> name and property name or by using an attribute to specify the field
>>>> sequence number.
>>>>>
>>>>> Is this something that would be of interest to the Avro project? My
>>>> company has approved committing the code and I’d be available and happy
>> to
>>>> maintain this code and work on other parts of the C# implementation.
>>>>>
>>>>> Regards.
>>>>>
>>>>>
>>>>
>>>>
>>>
>>> --
>>>
>>> [image: 51b630b05e01a6d5134ccfd520f547c4.png]
>>>
>>> Brian Lachniet
>>>
>>> Software Engineer
>>>
>>> E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
>>>
>>> <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
>>
>>
>
> --
>
> [image: 51b630b05e01a6d5134ccfd520f547c4.png]
>
> Brian Lachniet
>
> Software Engineer
>
> E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
>
> <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
Re: C# POCO serializer/deserializer
Posted by Brian Lachniet <bl...@gmail.com>.
Hi Patrick!
I'm sorry I haven't gotten back to the Reflect review yet. I'm hoping to
get back on that this weekend.
That sounds like a good list of additions. I'm particularly interested in
having support for JSON encoding/decoding in C#. I bet we could get a long
ways simply porting the JsonEncoder
<https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/io/JsonEncoder.java>
from the Java bindings.
I agree that we need to bump the Newtonsoft.Json verison. That just came up
in another issue recently. What version of Newtonsoft.Json do you think we
should upgrade to? I'm a little wary of bumping all the way to the latest,
as that would force everyone using Avro to also go to the latest version.
Maybe that's not a bad thing, I'm not sure.
I'm not familiar with Roslyn-based code generation. I'll do some Googling
on that (if you've got some good articles or starting points to share, I'd
appreciate that). Codegen currently uses the System.CodeDom API. Would
Roslyn-based code generation be a replacement for that, or am I
misunderstanding something?
Thank you for your patience so far, Patrick. I'm looking forward to these
new features too!
On Thu, Jul 11, 2019 at 6:14 PM Patrick Farry <pa...@gmail.com>
wrote:
> Hi Brian,
>
> Not meaning to hassle you (well maybe a little). I’ve got a few other
> enhancements we'd like to do as well as the Reflect code, and was hoping
> not to have to maintain a bunch of branches.
>
> Here are the planned changes:
> - update newtonsoft.json and add the json path whenever the schema parser
> reports an error - we have a number of large schemas and debugging them is
> proving difficult
> - Roslyn based schema generator from C# class
> - Reflect option added to codegen (possibly updating codegen to use Roslyn)
> - Serialize to json
>
> Thanks.
>
> > On May 2, 2019, at 6:22 PM, Brian Lachniet <bl...@gmail.com> wrote:
> >
> > Hey Patrick,
> >
> > This sounds very useful! I'd love to see this introduced to the C#
> library.
> >
> > On Thu, May 2, 2019 at 5:47 PM Ivan Greene <ig...@fanthreesixty.com>
> > wrote:
> >
> >> Patrick,
> >>
> >> This sounds as though it would be the C# equivalent of Java's
> ReflectData,
> >> which has been part of the Avro API for several years now, so it’s
> likely
> >> there would be interest. Your best bet is to open a Jira describing the
> >> planned feature and open a pull request on Github to start the
> discussion.
> >> The contributing page on the wiki is a bit out of date and still
> recommends
> >> submitting patches on Jira but I believe that Github is now the official
> >> repository.
> >>
> >>> On May 1, 2019, at 4:31 PM, Patrick Farry <pa...@gmail.com>
> >> wrote:
> >>>
> >>> Hi All,
> >>>
> >>> I have written code that implements Avro serialization for POCO classes
> >> - i.e. classes that are not generated by Avro codegen and do not
> implement
> >> ISpecificRecord. The idea was to make it work as much like JSON.net as
> >> possible.
> >>>
> >>> The serializer inherits from SpecificDefaultWriter and the deserializer
> >> from SpecificDefaultReader.
> >>>
> >>> Avro fields are mapped to C# properties either by matching the field
> >> name and property name or by using an attribute to specify the field
> >> sequence number.
> >>>
> >>> Is this something that would be of interest to the Avro project? My
> >> company has approved committing the code and I’d be available and happy
> to
> >> maintain this code and work on other parts of the C# implementation.
> >>>
> >>> Regards.
> >>>
> >>>
> >>
> >>
> >
> > --
> >
> > [image: 51b630b05e01a6d5134ccfd520f547c4.png]
> >
> > Brian Lachniet
> >
> > Software Engineer
> >
> > E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
> >
> > <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
>
>
--
[image: 51b630b05e01a6d5134ccfd520f547c4.png]
Brian Lachniet
Software Engineer
E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
<https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>