You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Driesprong, Fokko" <fo...@driesprong.frl> on 2019/04/30 10:54:28 UTC
[VOTE] Release Apache Avro 1.9.0 RC3
Hi everyone,
Since the last release of Apache Avro 1.8.2 on May 31, 2017. Two years
later,
I'm thrilled to propose the following RC to be released as official Apache
Avro 1.9.0 release.
The commit id is 24ff48c32d8d13433a1e31e485ef2af187d1eb62
* This corresponds to the tag: release-1.9.0-rc3
* https://github.com/apache/avro/releases/tag/release-1.9.0-rc3/
The release tarball, signature, and checksums are here:
* https://dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc3/
You can find the KEYS file here:
* https://dist.apache.org/repos/dist/dev/avro/KEYS
Binary artifacts for Java are staged in Nexus here:
*
https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.0/
This release includes 272 Jira issues:
https://issues.apache.org/jira/projects/AVRO/versions/12333394
* Deprecate Joda-Time in favor of Java8 JSR310 and setting it as default
* Remove support for Hadoop 1.x
* Move from Jackson 1.x to 2.9
* Add ZStandard Codec
* Lots of updates on the dependencies to fix CVE's
* and many, many more!
Since RC1, two commits have been added:
* https://jira.apache.org/jira/browse/AVRO-2381
* https://jira.apache.org/jira/browse/AVRO-2383
Since RC2 the SHA1/MD5 checksums have been replaced with SHA512
Please download, verify, and test. This vote will remain open for at least
72 hours. Given sufficient votes, I would like to close it on or about
midnight
on Saturday, 4th of May 2019.
[ ] +1 Release this as Apache Avro 1.9.0
[ ] +0
[ ] -1 Do not release this because...
Consider this a +1 (non-binding) from my side:
* Compiled the new version of Parquet against the Divolte collector and
Apache Parquet
Cheers, Fokko Driesprong
Re: C# POCO serializer/deserializer
Posted by Patrick Farry <pa...@gmail.com>.
Apologies but I had to make everything StyleCop compliant for work. Everything is basically the same but there are lot of spaces added/removed and a couple of new files because it doesn’t like multiple classes in a file. Makes it a pain to review.
> On Jul 11, 2019, at 5:05 PM, Brian Lachniet <bl...@gmail.com> wrote:
>
> Hi Patrick!
>
> I'm sorry I haven't gotten back to the Reflect review yet. I'm hoping to
> get back on that this weekend.
>
> That sounds like a good list of additions. I'm particularly interested in
> having support for JSON encoding/decoding in C#. I bet we could get a long
> ways simply porting the JsonEncoder
> <https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/io/JsonEncoder.java>
> from the Java bindings.
>
> I agree that we need to bump the Newtonsoft.Json verison. That just came up
> in another issue recently. What version of Newtonsoft.Json do you think we
> should upgrade to? I'm a little wary of bumping all the way to the latest,
> as that would force everyone using Avro to also go to the latest version.
> Maybe that's not a bad thing, I'm not sure.
>
> I'm not familiar with Roslyn-based code generation. I'll do some Googling
> on that (if you've got some good articles or starting points to share, I'd
> appreciate that). Codegen currently uses the System.CodeDom API. Would
> Roslyn-based code generation be a replacement for that, or am I
> misunderstanding something?
>
> Thank you for your patience so far, Patrick. I'm looking forward to these
> new features too!
>
> On Thu, Jul 11, 2019 at 6:14 PM Patrick Farry <pa...@gmail.com>
> wrote:
>
>> Hi Brian,
>>
>> Not meaning to hassle you (well maybe a little). I’ve got a few other
>> enhancements we'd like to do as well as the Reflect code, and was hoping
>> not to have to maintain a bunch of branches.
>>
>> Here are the planned changes:
>> - update newtonsoft.json and add the json path whenever the schema parser
>> reports an error - we have a number of large schemas and debugging them is
>> proving difficult
>> - Roslyn based schema generator from C# class
>> - Reflect option added to codegen (possibly updating codegen to use Roslyn)
>> - Serialize to json
>>
>> Thanks.
>>
>>> On May 2, 2019, at 6:22 PM, Brian Lachniet <bl...@gmail.com> wrote:
>>>
>>> Hey Patrick,
>>>
>>> This sounds very useful! I'd love to see this introduced to the C#
>> library.
>>>
>>> On Thu, May 2, 2019 at 5:47 PM Ivan Greene <ig...@fanthreesixty.com>
>>> wrote:
>>>
>>>> Patrick,
>>>>
>>>> This sounds as though it would be the C# equivalent of Java's
>> ReflectData,
>>>> which has been part of the Avro API for several years now, so it’s
>> likely
>>>> there would be interest. Your best bet is to open a Jira describing the
>>>> planned feature and open a pull request on Github to start the
>> discussion.
>>>> The contributing page on the wiki is a bit out of date and still
>> recommends
>>>> submitting patches on Jira but I believe that Github is now the official
>>>> repository.
>>>>
>>>>> On May 1, 2019, at 4:31 PM, Patrick Farry <pa...@gmail.com>
>>>> wrote:
>>>>>
>>>>> Hi All,
>>>>>
>>>>> I have written code that implements Avro serialization for POCO classes
>>>> - i.e. classes that are not generated by Avro codegen and do not
>> implement
>>>> ISpecificRecord. The idea was to make it work as much like JSON.net as
>>>> possible.
>>>>>
>>>>> The serializer inherits from SpecificDefaultWriter and the deserializer
>>>> from SpecificDefaultReader.
>>>>>
>>>>> Avro fields are mapped to C# properties either by matching the field
>>>> name and property name or by using an attribute to specify the field
>>>> sequence number.
>>>>>
>>>>> Is this something that would be of interest to the Avro project? My
>>>> company has approved committing the code and I’d be available and happy
>> to
>>>> maintain this code and work on other parts of the C# implementation.
>>>>>
>>>>> Regards.
>>>>>
>>>>>
>>>>
>>>>
>>>
>>> --
>>>
>>> [image: 51b630b05e01a6d5134ccfd520f547c4.png]
>>>
>>> Brian Lachniet
>>>
>>> Software Engineer
>>>
>>> E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
>>>
>>> <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
>>
>>
>
> --
>
> [image: 51b630b05e01a6d5134ccfd520f547c4.png]
>
> Brian Lachniet
>
> Software Engineer
>
> E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
>
> <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
Re: C# POCO serializer/deserializer
Posted by Patrick Farry <pa...@gmail.com>.
Hi Brian,
Thanks for the reply (and for shepherding the project).
Regarding Json. I’ll take a look at the Java code. I’ve also used Newtonsoft for things like this previously. JsonTextReader/JsonTextWriter look like they would map easily to the Avro Encode/Decoder interfaces. The banner on the Json.NET <http://json.net/> documentation says "JsonTextWriter Class Represents a writer that provides a fast, non-cached, forward-only way of generating JSON data.”, which seems like what we need. https://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_JsonTextWriter.htm <https://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_JsonTextWriter.htm>
Regarding Newtonsoft version. Maybe 10.0.3? It is 2 years old and 2 major versions back. It is also by far the most popular download on nuget. For what I’m trying to do (better error reporting while parsing) I just need the JToken to to have the Path property.
Regarding Roslyn. This is a detailed intro/tutorial https://medium.com/@CPP_Coder/introduction-to-roslyn-and-its-use-in-program-development-bce2043fc45d <https://medium.com/@CPP_Coder/introduction-to-roslyn-and-its-use-in-program-development-bce2043fc45d>. It is the compiler framework that’s being used for code analysis tools, cross compilers etc. For code->schema I’m thinking to do the following:
- open the solution
- use the syntax walker https://joshvarty.com/2014/07/26/learn-roslyn-now-part-4-csharpsyntaxwalker/ <https://joshvarty.com/2014/07/26/learn-roslyn-now-part-4-csharpsyntaxwalker/> to parse the required class, then generate a schema (or possibly a protocol for an entire namespace).
I haven’t thought a lot about the code generation side. But what I’m hoping is that we can parse the C#, as with schema generation, and then generate updates to the code by modifying the syntax tree. This will have the effect of editing the C# and preserving any edits, unmapped properties, and comments the developer has made. https://msdn.microsoft.com/en-us/magazine/mt808499.aspx <https://msdn.microsoft.com/en-us/magazine/mt808499.aspx>. As I’m typing this I think the reverse for the schema would be good too - load C# and schema and then edit the schema tree to sync any changes rather than just regenerating it every time.
> On Jul 11, 2019, at 5:05 PM, Brian Lachniet <bl...@gmail.com> wrote:
>
> Hi Patrick!
>
> I'm sorry I haven't gotten back to the Reflect review yet. I'm hoping to
> get back on that this weekend.
>
> That sounds like a good list of additions. I'm particularly interested in
> having support for JSON encoding/decoding in C#. I bet we could get a long
> ways simply porting the JsonEncoder
> <https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/io/JsonEncoder.java>
> from the Java bindings.
>
> I agree that we need to bump the Newtonsoft.Json verison. That just came up
> in another issue recently. What version of Newtonsoft.Json do you think we
> should upgrade to? I'm a little wary of bumping all the way to the latest,
> as that would force everyone using Avro to also go to the latest version.
> Maybe that's not a bad thing, I'm not sure.
>
> I'm not familiar with Roslyn-based code generation. I'll do some Googling
> on that (if you've got some good articles or starting points to share, I'd
> appreciate that). Codegen currently uses the System.CodeDom API. Would
> Roslyn-based code generation be a replacement for that, or am I
> misunderstanding something?
>
> Thank you for your patience so far, Patrick. I'm looking forward to these
> new features too!
>
> On Thu, Jul 11, 2019 at 6:14 PM Patrick Farry <pa...@gmail.com>
> wrote:
>
>> Hi Brian,
>>
>> Not meaning to hassle you (well maybe a little). I’ve got a few other
>> enhancements we'd like to do as well as the Reflect code, and was hoping
>> not to have to maintain a bunch of branches.
>>
>> Here are the planned changes:
>> - update newtonsoft.json and add the json path whenever the schema parser
>> reports an error - we have a number of large schemas and debugging them is
>> proving difficult
>> - Roslyn based schema generator from C# class
>> - Reflect option added to codegen (possibly updating codegen to use Roslyn)
>> - Serialize to json
>>
>> Thanks.
>>
>>> On May 2, 2019, at 6:22 PM, Brian Lachniet <bl...@gmail.com> wrote:
>>>
>>> Hey Patrick,
>>>
>>> This sounds very useful! I'd love to see this introduced to the C#
>> library.
>>>
>>> On Thu, May 2, 2019 at 5:47 PM Ivan Greene <ig...@fanthreesixty.com>
>>> wrote:
>>>
>>>> Patrick,
>>>>
>>>> This sounds as though it would be the C# equivalent of Java's
>> ReflectData,
>>>> which has been part of the Avro API for several years now, so it’s
>> likely
>>>> there would be interest. Your best bet is to open a Jira describing the
>>>> planned feature and open a pull request on Github to start the
>> discussion.
>>>> The contributing page on the wiki is a bit out of date and still
>> recommends
>>>> submitting patches on Jira but I believe that Github is now the official
>>>> repository.
>>>>
>>>>> On May 1, 2019, at 4:31 PM, Patrick Farry <pa...@gmail.com>
>>>> wrote:
>>>>>
>>>>> Hi All,
>>>>>
>>>>> I have written code that implements Avro serialization for POCO classes
>>>> - i.e. classes that are not generated by Avro codegen and do not
>> implement
>>>> ISpecificRecord. The idea was to make it work as much like JSON.net as
>>>> possible.
>>>>>
>>>>> The serializer inherits from SpecificDefaultWriter and the deserializer
>>>> from SpecificDefaultReader.
>>>>>
>>>>> Avro fields are mapped to C# properties either by matching the field
>>>> name and property name or by using an attribute to specify the field
>>>> sequence number.
>>>>>
>>>>> Is this something that would be of interest to the Avro project? My
>>>> company has approved committing the code and I’d be available and happy
>> to
>>>> maintain this code and work on other parts of the C# implementation.
>>>>>
>>>>> Regards.
>>>>>
>>>>>
>>>>
>>>>
>>>
>>> --
>>>
>>> [image: 51b630b05e01a6d5134ccfd520f547c4.png]
>>>
>>> Brian Lachniet
>>>
>>> Software Engineer
>>>
>>> E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
>>>
>>> <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
>>
>>
>
> --
>
> [image: 51b630b05e01a6d5134ccfd520f547c4.png]
>
> Brian Lachniet
>
> Software Engineer
>
> E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
>
> <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
Re: C# POCO serializer/deserializer
Posted by Patrick Farry <pa...@gmail.com>.
The schema generator is checked in as a separate project for now. It takes a number of C# classes and generates an Avro protocol using the Microsoft.CodeAnalysis package.
https://github.com/pa009fa/schemagen <https://github.com/pa009fa/schemagen>
There is a driver program which runs with the following parameters:
$ dotnet run -- --help
schemagen 1.0.0
Copyright (C) 2019 schemagen
-i, --include Required. Includes
-o, --outfile (Default: .) Output file. . for console
-t, --target Required. File name of the type to generate
-d, --defaultconverters Required. Default converters
-c, --converters Required. Default converters
--help Display this help screen.
--version Display version information.
The VSCode debugger is set up to generate a protocol from the classes in the csharp folder.
There are a number of new attributes to help when specifying the schema. Examples are in the Z.cs file. Attribute specify unions, default values etc.
The Reflect array helpers are not supported yet.
Any and all feedback is welcome.
At this stage I’m not sure it’s ready fro a detailed code review (but that would be fine as well if anyone has the time).
Also, any thoughts on how (whether?) to integrate it into the project are also welcome.
Cheers.
> On Aug 10, 2019, at 10:36 AM, Brian Lachniet <bl...@gmail.com> wrote:
>
> Hey Patrick,
>
> That sounds pretty cool. I think that the best way to handle this right
> now, if you believe there is additional work to make it ready for more
> general use, is to keep it on a separate branch in your repository.
>
> I wonder if this sort of functionality exists in other language bindings
> (I'm really only familiar with the C# bindings). For those familiar with
> the other languages, has this kind of thing been done before?
>
> I agree that the C# bindings could really benefit from having a JSON
> encoder. And yes, we should definitely take advantage of the JSON.net
> library. In implementation, I imagine this would take the form of new
> implementations of the *Encoder* and *Decoder* interfaces, *JsonEncoder *and
> *JsonDecoder*.
>
> On Mon, Aug 5, 2019 at 11:56 PM Patrick Farry <pa...@gmail.com>
> wrote:
>
>> Hi Brian,
>>
>> I got the schema from C# source code generator working, at least well
>> enough for us. Not sure it is ready for general use but if you’re
>> interested I can send the code. What is the usual practice for things under
>> development - should I create a PR even if the code is some way off from
>> being mergeable or just a branch of my repo?
>>
>> Our use case is that the developers write C# DTO classes and don’t need to
>> worry much about the schema. We build a protocol from the classes and then
>> schemas for each top level object from the protocol.
>>
>> Also, I think we (Pitney Bowes) need to get something going about a JSON
>> encoder. Shifting to Avro is a bit of a stretch for some of our developers
>> since they can’t easily see the message contents without additional
>> decoding tools. Are you ok with basing it on JSON.net? It should be pretty
>> fast to get something working.
>>
>>> On Jul 11, 2019, at 5:05 PM, Brian Lachniet <bl...@gmail.com> wrote:
>>>
>>> Hi Patrick!
>>>
>>> I'm sorry I haven't gotten back to the Reflect review yet. I'm hoping to
>>> get back on that this weekend.
>>>
>>> That sounds like a good list of additions. I'm particularly interested in
>>> having support for JSON encoding/decoding in C#. I bet we could get a
>> long
>>> ways simply porting the JsonEncoder
>>> <
>> https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/io/JsonEncoder.java
>>>
>>> from the Java bindings.
>>>
>>> I agree that we need to bump the Newtonsoft.Json verison. That just came
>> up
>>> in another issue recently. What version of Newtonsoft.Json do you think
>> we
>>> should upgrade to? I'm a little wary of bumping all the way to the
>> latest,
>>> as that would force everyone using Avro to also go to the latest version.
>>> Maybe that's not a bad thing, I'm not sure.
>>>
>>> I'm not familiar with Roslyn-based code generation. I'll do some Googling
>>> on that (if you've got some good articles or starting points to share,
>> I'd
>>> appreciate that). Codegen currently uses the System.CodeDom API. Would
>>> Roslyn-based code generation be a replacement for that, or am I
>>> misunderstanding something?
>>>
>>> Thank you for your patience so far, Patrick. I'm looking forward to these
>>> new features too!
>>>
>>> On Thu, Jul 11, 2019 at 6:14 PM Patrick Farry <patrick.s.farry@gmail.com
>>>
>>> wrote:
>>>
>>>> Hi Brian,
>>>>
>>>> Not meaning to hassle you (well maybe a little). I’ve got a few other
>>>> enhancements we'd like to do as well as the Reflect code, and was hoping
>>>> not to have to maintain a bunch of branches.
>>>>
>>>> Here are the planned changes:
>>>> - update newtonsoft.json and add the json path whenever the schema
>> parser
>>>> reports an error - we have a number of large schemas and debugging them
>> is
>>>> proving difficult
>>>> - Roslyn based schema generator from C# class
>>>> - Reflect option added to codegen (possibly updating codegen to use
>> Roslyn)
>>>> - Serialize to json
>>>>
>>>> Thanks.
>>>>
>>>>> On May 2, 2019, at 6:22 PM, Brian Lachniet <bl...@gmail.com>
>> wrote:
>>>>>
>>>>> Hey Patrick,
>>>>>
>>>>> This sounds very useful! I'd love to see this introduced to the C#
>>>> library.
>>>>>
>>>>> On Thu, May 2, 2019 at 5:47 PM Ivan Greene <ig...@fanthreesixty.com>
>>>>> wrote:
>>>>>
>>>>>> Patrick,
>>>>>>
>>>>>> This sounds as though it would be the C# equivalent of Java's
>>>> ReflectData,
>>>>>> which has been part of the Avro API for several years now, so it’s
>>>> likely
>>>>>> there would be interest. Your best bet is to open a Jira describing
>> the
>>>>>> planned feature and open a pull request on Github to start the
>>>> discussion.
>>>>>> The contributing page on the wiki is a bit out of date and still
>>>> recommends
>>>>>> submitting patches on Jira but I believe that Github is now the
>> official
>>>>>> repository.
>>>>>>
>>>>>>> On May 1, 2019, at 4:31 PM, Patrick Farry <patrick.s.farry@gmail.com
>>>
>>>>>> wrote:
>>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> I have written code that implements Avro serialization for POCO
>> classes
>>>>>> - i.e. classes that are not generated by Avro codegen and do not
>>>> implement
>>>>>> ISpecificRecord. The idea was to make it work as much like JSON.net as
>>>>>> possible.
>>>>>>>
>>>>>>> The serializer inherits from SpecificDefaultWriter and the
>> deserializer
>>>>>> from SpecificDefaultReader.
>>>>>>>
>>>>>>> Avro fields are mapped to C# properties either by matching the field
>>>>>> name and property name or by using an attribute to specify the field
>>>>>> sequence number.
>>>>>>>
>>>>>>> Is this something that would be of interest to the Avro project? My
>>>>>> company has approved committing the code and I’d be available and
>> happy
>>>> to
>>>>>> maintain this code and work on other parts of the C# implementation.
>>>>>>>
>>>>>>> Regards.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> [image: 51b630b05e01a6d5134ccfd520f547c4.png]
>>>>>
>>>>> Brian Lachniet
>>>>>
>>>>> Software Engineer
>>>>>
>>>>> E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
>>>>>
>>>>> <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
>>>>
>>>>
>>>
>>> --
>>>
>>> [image: 51b630b05e01a6d5134ccfd520f547c4.png]
>>>
>>> Brian Lachniet
>>>
>>> Software Engineer
>>>
>>> E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
>>>
>>> <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
>>
>>
>
> --
>
> [image: 51b630b05e01a6d5134ccfd520f547c4.png]
>
> Brian Lachniet
>
> Software Engineer
>
> E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
>
> <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
Re: C# POCO serializer/deserializer
Posted by Brian Lachniet <bl...@gmail.com>.
Hey Patrick,
That sounds pretty cool. I think that the best way to handle this right
now, if you believe there is additional work to make it ready for more
general use, is to keep it on a separate branch in your repository.
I wonder if this sort of functionality exists in other language bindings
(I'm really only familiar with the C# bindings). For those familiar with
the other languages, has this kind of thing been done before?
I agree that the C# bindings could really benefit from having a JSON
encoder. And yes, we should definitely take advantage of the JSON.net
library. In implementation, I imagine this would take the form of new
implementations of the *Encoder* and *Decoder* interfaces, *JsonEncoder *and
*JsonDecoder*.
On Mon, Aug 5, 2019 at 11:56 PM Patrick Farry <pa...@gmail.com>
wrote:
> Hi Brian,
>
> I got the schema from C# source code generator working, at least well
> enough for us. Not sure it is ready for general use but if you’re
> interested I can send the code. What is the usual practice for things under
> development - should I create a PR even if the code is some way off from
> being mergeable or just a branch of my repo?
>
> Our use case is that the developers write C# DTO classes and don’t need to
> worry much about the schema. We build a protocol from the classes and then
> schemas for each top level object from the protocol.
>
> Also, I think we (Pitney Bowes) need to get something going about a JSON
> encoder. Shifting to Avro is a bit of a stretch for some of our developers
> since they can’t easily see the message contents without additional
> decoding tools. Are you ok with basing it on JSON.net? It should be pretty
> fast to get something working.
>
> > On Jul 11, 2019, at 5:05 PM, Brian Lachniet <bl...@gmail.com> wrote:
> >
> > Hi Patrick!
> >
> > I'm sorry I haven't gotten back to the Reflect review yet. I'm hoping to
> > get back on that this weekend.
> >
> > That sounds like a good list of additions. I'm particularly interested in
> > having support for JSON encoding/decoding in C#. I bet we could get a
> long
> > ways simply porting the JsonEncoder
> > <
> https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/io/JsonEncoder.java
> >
> > from the Java bindings.
> >
> > I agree that we need to bump the Newtonsoft.Json verison. That just came
> up
> > in another issue recently. What version of Newtonsoft.Json do you think
> we
> > should upgrade to? I'm a little wary of bumping all the way to the
> latest,
> > as that would force everyone using Avro to also go to the latest version.
> > Maybe that's not a bad thing, I'm not sure.
> >
> > I'm not familiar with Roslyn-based code generation. I'll do some Googling
> > on that (if you've got some good articles or starting points to share,
> I'd
> > appreciate that). Codegen currently uses the System.CodeDom API. Would
> > Roslyn-based code generation be a replacement for that, or am I
> > misunderstanding something?
> >
> > Thank you for your patience so far, Patrick. I'm looking forward to these
> > new features too!
> >
> > On Thu, Jul 11, 2019 at 6:14 PM Patrick Farry <patrick.s.farry@gmail.com
> >
> > wrote:
> >
> >> Hi Brian,
> >>
> >> Not meaning to hassle you (well maybe a little). I’ve got a few other
> >> enhancements we'd like to do as well as the Reflect code, and was hoping
> >> not to have to maintain a bunch of branches.
> >>
> >> Here are the planned changes:
> >> - update newtonsoft.json and add the json path whenever the schema
> parser
> >> reports an error - we have a number of large schemas and debugging them
> is
> >> proving difficult
> >> - Roslyn based schema generator from C# class
> >> - Reflect option added to codegen (possibly updating codegen to use
> Roslyn)
> >> - Serialize to json
> >>
> >> Thanks.
> >>
> >>> On May 2, 2019, at 6:22 PM, Brian Lachniet <bl...@gmail.com>
> wrote:
> >>>
> >>> Hey Patrick,
> >>>
> >>> This sounds very useful! I'd love to see this introduced to the C#
> >> library.
> >>>
> >>> On Thu, May 2, 2019 at 5:47 PM Ivan Greene <ig...@fanthreesixty.com>
> >>> wrote:
> >>>
> >>>> Patrick,
> >>>>
> >>>> This sounds as though it would be the C# equivalent of Java's
> >> ReflectData,
> >>>> which has been part of the Avro API for several years now, so it’s
> >> likely
> >>>> there would be interest. Your best bet is to open a Jira describing
> the
> >>>> planned feature and open a pull request on Github to start the
> >> discussion.
> >>>> The contributing page on the wiki is a bit out of date and still
> >> recommends
> >>>> submitting patches on Jira but I believe that Github is now the
> official
> >>>> repository.
> >>>>
> >>>>> On May 1, 2019, at 4:31 PM, Patrick Farry <patrick.s.farry@gmail.com
> >
> >>>> wrote:
> >>>>>
> >>>>> Hi All,
> >>>>>
> >>>>> I have written code that implements Avro serialization for POCO
> classes
> >>>> - i.e. classes that are not generated by Avro codegen and do not
> >> implement
> >>>> ISpecificRecord. The idea was to make it work as much like JSON.net as
> >>>> possible.
> >>>>>
> >>>>> The serializer inherits from SpecificDefaultWriter and the
> deserializer
> >>>> from SpecificDefaultReader.
> >>>>>
> >>>>> Avro fields are mapped to C# properties either by matching the field
> >>>> name and property name or by using an attribute to specify the field
> >>>> sequence number.
> >>>>>
> >>>>> Is this something that would be of interest to the Avro project? My
> >>>> company has approved committing the code and I’d be available and
> happy
> >> to
> >>>> maintain this code and work on other parts of the C# implementation.
> >>>>>
> >>>>> Regards.
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>> --
> >>>
> >>> [image: 51b630b05e01a6d5134ccfd520f547c4.png]
> >>>
> >>> Brian Lachniet
> >>>
> >>> Software Engineer
> >>>
> >>> E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
> >>>
> >>> <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
> >>
> >>
> >
> > --
> >
> > [image: 51b630b05e01a6d5134ccfd520f547c4.png]
> >
> > Brian Lachniet
> >
> > Software Engineer
> >
> > E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
> >
> > <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
>
>
--
[image: 51b630b05e01a6d5134ccfd520f547c4.png]
Brian Lachniet
Software Engineer
E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
<https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
Re: C# POCO serializer/deserializer
Posted by Patrick Farry <pa...@gmail.com>.
Hi Brian,
I got the schema from C# source code generator working, at least well enough for us. Not sure it is ready for general use but if you’re interested I can send the code. What is the usual practice for things under development - should I create a PR even if the code is some way off from being mergeable or just a branch of my repo?
Our use case is that the developers write C# DTO classes and don’t need to worry much about the schema. We build a protocol from the classes and then schemas for each top level object from the protocol.
Also, I think we (Pitney Bowes) need to get something going about a JSON encoder. Shifting to Avro is a bit of a stretch for some of our developers since they can’t easily see the message contents without additional decoding tools. Are you ok with basing it on JSON.net? It should be pretty fast to get something working.
> On Jul 11, 2019, at 5:05 PM, Brian Lachniet <bl...@gmail.com> wrote:
>
> Hi Patrick!
>
> I'm sorry I haven't gotten back to the Reflect review yet. I'm hoping to
> get back on that this weekend.
>
> That sounds like a good list of additions. I'm particularly interested in
> having support for JSON encoding/decoding in C#. I bet we could get a long
> ways simply porting the JsonEncoder
> <https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/io/JsonEncoder.java>
> from the Java bindings.
>
> I agree that we need to bump the Newtonsoft.Json verison. That just came up
> in another issue recently. What version of Newtonsoft.Json do you think we
> should upgrade to? I'm a little wary of bumping all the way to the latest,
> as that would force everyone using Avro to also go to the latest version.
> Maybe that's not a bad thing, I'm not sure.
>
> I'm not familiar with Roslyn-based code generation. I'll do some Googling
> on that (if you've got some good articles or starting points to share, I'd
> appreciate that). Codegen currently uses the System.CodeDom API. Would
> Roslyn-based code generation be a replacement for that, or am I
> misunderstanding something?
>
> Thank you for your patience so far, Patrick. I'm looking forward to these
> new features too!
>
> On Thu, Jul 11, 2019 at 6:14 PM Patrick Farry <pa...@gmail.com>
> wrote:
>
>> Hi Brian,
>>
>> Not meaning to hassle you (well maybe a little). I’ve got a few other
>> enhancements we'd like to do as well as the Reflect code, and was hoping
>> not to have to maintain a bunch of branches.
>>
>> Here are the planned changes:
>> - update newtonsoft.json and add the json path whenever the schema parser
>> reports an error - we have a number of large schemas and debugging them is
>> proving difficult
>> - Roslyn based schema generator from C# class
>> - Reflect option added to codegen (possibly updating codegen to use Roslyn)
>> - Serialize to json
>>
>> Thanks.
>>
>>> On May 2, 2019, at 6:22 PM, Brian Lachniet <bl...@gmail.com> wrote:
>>>
>>> Hey Patrick,
>>>
>>> This sounds very useful! I'd love to see this introduced to the C#
>> library.
>>>
>>> On Thu, May 2, 2019 at 5:47 PM Ivan Greene <ig...@fanthreesixty.com>
>>> wrote:
>>>
>>>> Patrick,
>>>>
>>>> This sounds as though it would be the C# equivalent of Java's
>> ReflectData,
>>>> which has been part of the Avro API for several years now, so it’s
>> likely
>>>> there would be interest. Your best bet is to open a Jira describing the
>>>> planned feature and open a pull request on Github to start the
>> discussion.
>>>> The contributing page on the wiki is a bit out of date and still
>> recommends
>>>> submitting patches on Jira but I believe that Github is now the official
>>>> repository.
>>>>
>>>>> On May 1, 2019, at 4:31 PM, Patrick Farry <pa...@gmail.com>
>>>> wrote:
>>>>>
>>>>> Hi All,
>>>>>
>>>>> I have written code that implements Avro serialization for POCO classes
>>>> - i.e. classes that are not generated by Avro codegen and do not
>> implement
>>>> ISpecificRecord. The idea was to make it work as much like JSON.net as
>>>> possible.
>>>>>
>>>>> The serializer inherits from SpecificDefaultWriter and the deserializer
>>>> from SpecificDefaultReader.
>>>>>
>>>>> Avro fields are mapped to C# properties either by matching the field
>>>> name and property name or by using an attribute to specify the field
>>>> sequence number.
>>>>>
>>>>> Is this something that would be of interest to the Avro project? My
>>>> company has approved committing the code and I’d be available and happy
>> to
>>>> maintain this code and work on other parts of the C# implementation.
>>>>>
>>>>> Regards.
>>>>>
>>>>>
>>>>
>>>>
>>>
>>> --
>>>
>>> [image: 51b630b05e01a6d5134ccfd520f547c4.png]
>>>
>>> Brian Lachniet
>>>
>>> Software Engineer
>>>
>>> E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
>>>
>>> <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
>>
>>
>
> --
>
> [image: 51b630b05e01a6d5134ccfd520f547c4.png]
>
> Brian Lachniet
>
> Software Engineer
>
> E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
>
> <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
Re: C# POCO serializer/deserializer
Posted by Brian Lachniet <bl...@gmail.com>.
Hi Patrick!
I'm sorry I haven't gotten back to the Reflect review yet. I'm hoping to
get back on that this weekend.
That sounds like a good list of additions. I'm particularly interested in
having support for JSON encoding/decoding in C#. I bet we could get a long
ways simply porting the JsonEncoder
<https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/io/JsonEncoder.java>
from the Java bindings.
I agree that we need to bump the Newtonsoft.Json verison. That just came up
in another issue recently. What version of Newtonsoft.Json do you think we
should upgrade to? I'm a little wary of bumping all the way to the latest,
as that would force everyone using Avro to also go to the latest version.
Maybe that's not a bad thing, I'm not sure.
I'm not familiar with Roslyn-based code generation. I'll do some Googling
on that (if you've got some good articles or starting points to share, I'd
appreciate that). Codegen currently uses the System.CodeDom API. Would
Roslyn-based code generation be a replacement for that, or am I
misunderstanding something?
Thank you for your patience so far, Patrick. I'm looking forward to these
new features too!
On Thu, Jul 11, 2019 at 6:14 PM Patrick Farry <pa...@gmail.com>
wrote:
> Hi Brian,
>
> Not meaning to hassle you (well maybe a little). I’ve got a few other
> enhancements we'd like to do as well as the Reflect code, and was hoping
> not to have to maintain a bunch of branches.
>
> Here are the planned changes:
> - update newtonsoft.json and add the json path whenever the schema parser
> reports an error - we have a number of large schemas and debugging them is
> proving difficult
> - Roslyn based schema generator from C# class
> - Reflect option added to codegen (possibly updating codegen to use Roslyn)
> - Serialize to json
>
> Thanks.
>
> > On May 2, 2019, at 6:22 PM, Brian Lachniet <bl...@gmail.com> wrote:
> >
> > Hey Patrick,
> >
> > This sounds very useful! I'd love to see this introduced to the C#
> library.
> >
> > On Thu, May 2, 2019 at 5:47 PM Ivan Greene <ig...@fanthreesixty.com>
> > wrote:
> >
> >> Patrick,
> >>
> >> This sounds as though it would be the C# equivalent of Java's
> ReflectData,
> >> which has been part of the Avro API for several years now, so it’s
> likely
> >> there would be interest. Your best bet is to open a Jira describing the
> >> planned feature and open a pull request on Github to start the
> discussion.
> >> The contributing page on the wiki is a bit out of date and still
> recommends
> >> submitting patches on Jira but I believe that Github is now the official
> >> repository.
> >>
> >>> On May 1, 2019, at 4:31 PM, Patrick Farry <pa...@gmail.com>
> >> wrote:
> >>>
> >>> Hi All,
> >>>
> >>> I have written code that implements Avro serialization for POCO classes
> >> - i.e. classes that are not generated by Avro codegen and do not
> implement
> >> ISpecificRecord. The idea was to make it work as much like JSON.net as
> >> possible.
> >>>
> >>> The serializer inherits from SpecificDefaultWriter and the deserializer
> >> from SpecificDefaultReader.
> >>>
> >>> Avro fields are mapped to C# properties either by matching the field
> >> name and property name or by using an attribute to specify the field
> >> sequence number.
> >>>
> >>> Is this something that would be of interest to the Avro project? My
> >> company has approved committing the code and I’d be available and happy
> to
> >> maintain this code and work on other parts of the C# implementation.
> >>>
> >>> Regards.
> >>>
> >>>
> >>
> >>
> >
> > --
> >
> > [image: 51b630b05e01a6d5134ccfd520f547c4.png]
> >
> > Brian Lachniet
> >
> > Software Engineer
> >
> > E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
> >
> > <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
>
>
--
[image: 51b630b05e01a6d5134ccfd520f547c4.png]
Brian Lachniet
Software Engineer
E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
<https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
Re: C# POCO serializer/deserializer
Posted by Patrick Farry <pa...@gmail.com>.
Hi Brian,
Not meaning to hassle you (well maybe a little). I’ve got a few other enhancements we'd like to do as well as the Reflect code, and was hoping not to have to maintain a bunch of branches.
Here are the planned changes:
- update newtonsoft.json and add the json path whenever the schema parser reports an error - we have a number of large schemas and debugging them is proving difficult
- Roslyn based schema generator from C# class
- Reflect option added to codegen (possibly updating codegen to use Roslyn)
- Serialize to json
Thanks.
> On May 2, 2019, at 6:22 PM, Brian Lachniet <bl...@gmail.com> wrote:
>
> Hey Patrick,
>
> This sounds very useful! I'd love to see this introduced to the C# library.
>
> On Thu, May 2, 2019 at 5:47 PM Ivan Greene <ig...@fanthreesixty.com>
> wrote:
>
>> Patrick,
>>
>> This sounds as though it would be the C# equivalent of Java's ReflectData,
>> which has been part of the Avro API for several years now, so it’s likely
>> there would be interest. Your best bet is to open a Jira describing the
>> planned feature and open a pull request on Github to start the discussion.
>> The contributing page on the wiki is a bit out of date and still recommends
>> submitting patches on Jira but I believe that Github is now the official
>> repository.
>>
>>> On May 1, 2019, at 4:31 PM, Patrick Farry <pa...@gmail.com>
>> wrote:
>>>
>>> Hi All,
>>>
>>> I have written code that implements Avro serialization for POCO classes
>> - i.e. classes that are not generated by Avro codegen and do not implement
>> ISpecificRecord. The idea was to make it work as much like JSON.net as
>> possible.
>>>
>>> The serializer inherits from SpecificDefaultWriter and the deserializer
>> from SpecificDefaultReader.
>>>
>>> Avro fields are mapped to C# properties either by matching the field
>> name and property name or by using an attribute to specify the field
>> sequence number.
>>>
>>> Is this something that would be of interest to the Avro project? My
>> company has approved committing the code and I’d be available and happy to
>> maintain this code and work on other parts of the C# implementation.
>>>
>>> Regards.
>>>
>>>
>>
>>
>
> --
>
> [image: 51b630b05e01a6d5134ccfd520f547c4.png]
>
> Brian Lachniet
>
> Software Engineer
>
> E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
>
> <https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
Re: C# POCO serializer/deserializer
Posted by Brian Lachniet <bl...@gmail.com>.
Hey Patrick,
This sounds very useful! I'd love to see this introduced to the C# library.
On Thu, May 2, 2019 at 5:47 PM Ivan Greene <ig...@fanthreesixty.com>
wrote:
> Patrick,
>
> This sounds as though it would be the C# equivalent of Java's ReflectData,
> which has been part of the Avro API for several years now, so it’s likely
> there would be interest. Your best bet is to open a Jira describing the
> planned feature and open a pull request on Github to start the discussion.
> The contributing page on the wiki is a bit out of date and still recommends
> submitting patches on Jira but I believe that Github is now the official
> repository.
>
> > On May 1, 2019, at 4:31 PM, Patrick Farry <pa...@gmail.com>
> wrote:
> >
> > Hi All,
> >
> > I have written code that implements Avro serialization for POCO classes
> - i.e. classes that are not generated by Avro codegen and do not implement
> ISpecificRecord. The idea was to make it work as much like JSON.net as
> possible.
> >
> > The serializer inherits from SpecificDefaultWriter and the deserializer
> from SpecificDefaultReader.
> >
> > Avro fields are mapped to C# properties either by matching the field
> name and property name or by using an attribute to specify the field
> sequence number.
> >
> > Is this something that would be of interest to the Avro project? My
> company has approved committing the code and I’d be available and happy to
> maintain this code and work on other parts of the C# implementation.
> >
> > Regards.
> >
> >
>
>
--
[image: 51b630b05e01a6d5134ccfd520f547c4.png]
Brian Lachniet
Software Engineer
E: blachniet@gmail.com | blachniet.com <http://www.blachniet.com>
<https://twitter.com/blachniet> <http://www.linkedin.com/in/blachniet>
Re: C# POCO serializer/deserializer
Posted by Ivan Greene <ig...@fanthreesixty.com>.
Patrick,
This sounds as though it would be the C# equivalent of Java's ReflectData, which has been part of the Avro API for several years now, so it’s likely there would be interest. Your best bet is to open a Jira describing the planned feature and open a pull request on Github to start the discussion. The contributing page on the wiki is a bit out of date and still recommends submitting patches on Jira but I believe that Github is now the official repository.
> On May 1, 2019, at 4:31 PM, Patrick Farry <pa...@gmail.com> wrote:
>
> Hi All,
>
> I have written code that implements Avro serialization for POCO classes - i.e. classes that are not generated by Avro codegen and do not implement ISpecificRecord. The idea was to make it work as much like JSON.net as possible.
>
> The serializer inherits from SpecificDefaultWriter and the deserializer from SpecificDefaultReader.
>
> Avro fields are mapped to C# properties either by matching the field name and property name or by using an attribute to specify the field sequence number.
>
> Is this something that would be of interest to the Avro project? My company has approved committing the code and I’d be available and happy to maintain this code and work on other parts of the C# implementation.
>
> Regards.
>
>
C# POCO serializer/deserializer
Posted by Patrick Farry <pa...@gmail.com>.
Hi All,
I have written code that implements Avro serialization for POCO classes - i.e. classes that are not generated by Avro codegen and do not implement ISpecificRecord. The idea was to make it work as much like JSON.net as possible.
The serializer inherits from SpecificDefaultWriter and the deserializer from SpecificDefaultReader.
Avro fields are mapped to C# properties either by matching the field name and property name or by using an attribute to specify the field sequence number.
Is this something that would be of interest to the Avro project? My company has approved committing the code and I’d be available and happy to maintain this code and work on other parts of the C# implementation.
Regards.
Re: [VOTE] Release Apache Avro 1.9.0 RC3
Posted by Ismaël Mejía <ie...@gmail.com>.
I am referring to the 'SNAPSHOT' suffix in the names (and content) of
the dist link.
https://dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc3/
It seems the github link is ok, but I assume that the 'defacto
version' for validation is the dist.apache.org one.
The binary staged artifacts look ok (the poms don't have the SNAPSHOT
suffix) so they should ok to test.
https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.0/
So maybe it is just a question of updating the files in dist.
Extra comment for the relase info 'avro-core' is now !MB smaller due
to the removal of the extra dependencies.
On Wed, May 1, 2019 at 10:39 PM Driesprong, Fokko <fo...@driesprong.frl> wrote:
>
> Thank you Ismaël for looking into it and for the additional interesting
> info.
>
> Are you referring to the SNAPSHOT in the tests? I can remove these:
> root@83c927afb89b:/avro# grep -R "SNAPSHOT" .
> ./lang/js/node_modules/uglify-js/tools/domprops.json:
> "ORDERED_NODE_SNAPSHOT_TYPE",
> ./lang/js/node_modules/uglify-js/tools/domprops.json:
> "UNORDERED_NODE_SNAPSHOT_TYPE",
> ./lang/java/archetypes/avro-service-archetype/src/test/integration/projects/basic/archetype.properties:version=0.1-SNAPSHOT
> ./lang/java/maven-plugin/src/test/resources/unit/idl/pom-jsr310.xml:
> <version>1.9.0-SNAPSHOT</version>
> ./lang/java/maven-plugin/src/test/resources/unit/idl/pom-joda.xml:
> <version>1.9.0-SNAPSHOT</version>
> ./lang/java/maven-plugin/src/test/resources/unit/protocol/pom-jsr310.xml:
> <version>1.9.0-SNAPSHOT</version>
> ./lang/java/maven-plugin/src/test/resources/unit/protocol/pom-joda.xml:
> <version>1.9.0-SNAPSHOT</version>
> ./lang/java/maven-plugin/src/test/resources/unit/schema/pom-jsr310.xml:
> <version>1.9.0-SNAPSHOT</version>
> ./lang/java/maven-plugin/src/test/resources/unit/schema/pom-joda.xml:
> <version>1.9.0-SNAPSHOT</version>
> ./lang/ruby/Rakefile:VERSION =
> File.open('../../share/VERSION.txt').read.sub('-SNAPSHOT', '.pre1').chomp
> ./lang/ruby/.gem/gems/json_pure-2.1.0/data/prototype.js: null,
> XPathResult.ORDERED_NODE_SNAPSHOT_TYPE, null);
> ./doc/examples/java-example/pom.xml: <version>1.0-SNAPSHOT</version>
>
> These have slipped the `mvn versions:set`.
>
> Cheers, Fokko
>
> Op di 30 apr. 2019 om 17:24 schreef Ismaël Mejía <ie...@gmail.com>:
>
> > I did a quick review on the JIRA issues included and extracted some
> > extra interesting info worth of addint to the release notes. Up to you
> > to choose which matter or not to be added.
> >
> > * Remove Jackson classes from public API
> > * Avro is built by default with Java 8
> > * Avro is compiled and tested too with Java 11 to guarantee compatibility
> > * Avro MapReduce is also compiled and tested with Hadoop 3
> > * Avro is now leaner, multiple dependencies were removed: guava,
> > paranamer, commons-codec and commons-logging
> > * Introduce JMH Performance Testing Framework
> > * Add Snappy support for C++ DataFile
> >
> > On Tue, Apr 30, 2019 at 4:50 PM Ismaël Mejía <ie...@gmail.com> wrote:
> > >
> > > Sorry, playing the killjoy again.
> > >
> > > It seems the files (and more critical the poms) still have the -SNAPSHOT
> > suffix.
> > > Also the comment of Dan Kulp about the extra directory structure in
> > > the main file build/avro-1.9.0 directory would be a nice extra thing
> > > to fix.
> > >
> > > On Tue, Apr 30, 2019 at 12:54 PM Driesprong, Fokko <fo...@driesprong.frl>
> > wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > Since the last release of Apache Avro 1.8.2 on May 31, 2017. Two years
> > > > later,
> > > > I'm thrilled to propose the following RC to be released as official
> > Apache
> > > > Avro 1.9.0 release.
> > > >
> > > > The commit id is 24ff48c32d8d13433a1e31e485ef2af187d1eb62
> > > > * This corresponds to the tag: release-1.9.0-rc3
> > > > * https://github.com/apache/avro/releases/tag/release-1.9.0-rc3/
> > > >
> > > > The release tarball, signature, and checksums are here:
> > > > * https://dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc3/
> > > >
> > > > You can find the KEYS file here:
> > > > * https://dist.apache.org/repos/dist/dev/avro/KEYS
> > > >
> > > > Binary artifacts for Java are staged in Nexus here:
> > > > *
> > > >
> > https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.0/
> > > >
> > > > This release includes 272 Jira issues:
> > > > https://issues.apache.org/jira/projects/AVRO/versions/12333394
> > > > * Deprecate Joda-Time in favor of Java8 JSR310 and setting it as
> > default
> > > > * Remove support for Hadoop 1.x
> > > > * Move from Jackson 1.x to 2.9
> > > > * Add ZStandard Codec
> > > > * Lots of updates on the dependencies to fix CVE's
> > > > * and many, many more!
> > > >
> > > > Since RC1, two commits have been added:
> > > > * https://jira.apache.org/jira/browse/AVRO-2381
> > > > * https://jira.apache.org/jira/browse/AVRO-2383
> > > >
> > > > Since RC2 the SHA1/MD5 checksums have been replaced with SHA512
> > > >
> > > > Please download, verify, and test. This vote will remain open for at
> > least
> > > > 72 hours. Given sufficient votes, I would like to close it on or about
> > > > midnight
> > > > on Saturday, 4th of May 2019.
> > > >
> > > > [ ] +1 Release this as Apache Avro 1.9.0
> > > > [ ] +0
> > > > [ ] -1 Do not release this because...
> > > >
> > > > Consider this a +1 (non-binding) from my side:
> > > > * Compiled the new version of Parquet against the Divolte collector and
> > > > Apache Parquet
> > > >
> > > > Cheers, Fokko Driesprong
> >
Re: [VOTE] Release Apache Avro 1.9.0 RC3
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Thank you Ismaël for looking into it and for the additional interesting
info.
Are you referring to the SNAPSHOT in the tests? I can remove these:
root@83c927afb89b:/avro# grep -R "SNAPSHOT" .
./lang/js/node_modules/uglify-js/tools/domprops.json:
"ORDERED_NODE_SNAPSHOT_TYPE",
./lang/js/node_modules/uglify-js/tools/domprops.json:
"UNORDERED_NODE_SNAPSHOT_TYPE",
./lang/java/archetypes/avro-service-archetype/src/test/integration/projects/basic/archetype.properties:version=0.1-SNAPSHOT
./lang/java/maven-plugin/src/test/resources/unit/idl/pom-jsr310.xml:
<version>1.9.0-SNAPSHOT</version>
./lang/java/maven-plugin/src/test/resources/unit/idl/pom-joda.xml:
<version>1.9.0-SNAPSHOT</version>
./lang/java/maven-plugin/src/test/resources/unit/protocol/pom-jsr310.xml:
<version>1.9.0-SNAPSHOT</version>
./lang/java/maven-plugin/src/test/resources/unit/protocol/pom-joda.xml:
<version>1.9.0-SNAPSHOT</version>
./lang/java/maven-plugin/src/test/resources/unit/schema/pom-jsr310.xml:
<version>1.9.0-SNAPSHOT</version>
./lang/java/maven-plugin/src/test/resources/unit/schema/pom-joda.xml:
<version>1.9.0-SNAPSHOT</version>
./lang/ruby/Rakefile:VERSION =
File.open('../../share/VERSION.txt').read.sub('-SNAPSHOT', '.pre1').chomp
./lang/ruby/.gem/gems/json_pure-2.1.0/data/prototype.js: null,
XPathResult.ORDERED_NODE_SNAPSHOT_TYPE, null);
./doc/examples/java-example/pom.xml: <version>1.0-SNAPSHOT</version>
These have slipped the `mvn versions:set`.
Cheers, Fokko
Op di 30 apr. 2019 om 17:24 schreef Ismaël Mejía <ie...@gmail.com>:
> I did a quick review on the JIRA issues included and extracted some
> extra interesting info worth of addint to the release notes. Up to you
> to choose which matter or not to be added.
>
> * Remove Jackson classes from public API
> * Avro is built by default with Java 8
> * Avro is compiled and tested too with Java 11 to guarantee compatibility
> * Avro MapReduce is also compiled and tested with Hadoop 3
> * Avro is now leaner, multiple dependencies were removed: guava,
> paranamer, commons-codec and commons-logging
> * Introduce JMH Performance Testing Framework
> * Add Snappy support for C++ DataFile
>
> On Tue, Apr 30, 2019 at 4:50 PM Ismaël Mejía <ie...@gmail.com> wrote:
> >
> > Sorry, playing the killjoy again.
> >
> > It seems the files (and more critical the poms) still have the -SNAPSHOT
> suffix.
> > Also the comment of Dan Kulp about the extra directory structure in
> > the main file build/avro-1.9.0 directory would be a nice extra thing
> > to fix.
> >
> > On Tue, Apr 30, 2019 at 12:54 PM Driesprong, Fokko <fo...@driesprong.frl>
> wrote:
> > >
> > > Hi everyone,
> > >
> > > Since the last release of Apache Avro 1.8.2 on May 31, 2017. Two years
> > > later,
> > > I'm thrilled to propose the following RC to be released as official
> Apache
> > > Avro 1.9.0 release.
> > >
> > > The commit id is 24ff48c32d8d13433a1e31e485ef2af187d1eb62
> > > * This corresponds to the tag: release-1.9.0-rc3
> > > * https://github.com/apache/avro/releases/tag/release-1.9.0-rc3/
> > >
> > > The release tarball, signature, and checksums are here:
> > > * https://dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc3/
> > >
> > > You can find the KEYS file here:
> > > * https://dist.apache.org/repos/dist/dev/avro/KEYS
> > >
> > > Binary artifacts for Java are staged in Nexus here:
> > > *
> > >
> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.0/
> > >
> > > This release includes 272 Jira issues:
> > > https://issues.apache.org/jira/projects/AVRO/versions/12333394
> > > * Deprecate Joda-Time in favor of Java8 JSR310 and setting it as
> default
> > > * Remove support for Hadoop 1.x
> > > * Move from Jackson 1.x to 2.9
> > > * Add ZStandard Codec
> > > * Lots of updates on the dependencies to fix CVE's
> > > * and many, many more!
> > >
> > > Since RC1, two commits have been added:
> > > * https://jira.apache.org/jira/browse/AVRO-2381
> > > * https://jira.apache.org/jira/browse/AVRO-2383
> > >
> > > Since RC2 the SHA1/MD5 checksums have been replaced with SHA512
> > >
> > > Please download, verify, and test. This vote will remain open for at
> least
> > > 72 hours. Given sufficient votes, I would like to close it on or about
> > > midnight
> > > on Saturday, 4th of May 2019.
> > >
> > > [ ] +1 Release this as Apache Avro 1.9.0
> > > [ ] +0
> > > [ ] -1 Do not release this because...
> > >
> > > Consider this a +1 (non-binding) from my side:
> > > * Compiled the new version of Parquet against the Divolte collector and
> > > Apache Parquet
> > >
> > > Cheers, Fokko Driesprong
>
Re: [VOTE] Release Apache Avro 1.9.0 RC3
Posted by Ismaël Mejía <ie...@gmail.com>.
I did a quick review on the JIRA issues included and extracted some
extra interesting info worth of addint to the release notes. Up to you
to choose which matter or not to be added.
* Remove Jackson classes from public API
* Avro is built by default with Java 8
* Avro is compiled and tested too with Java 11 to guarantee compatibility
* Avro MapReduce is also compiled and tested with Hadoop 3
* Avro is now leaner, multiple dependencies were removed: guava,
paranamer, commons-codec and commons-logging
* Introduce JMH Performance Testing Framework
* Add Snappy support for C++ DataFile
On Tue, Apr 30, 2019 at 4:50 PM Ismaël Mejía <ie...@gmail.com> wrote:
>
> Sorry, playing the killjoy again.
>
> It seems the files (and more critical the poms) still have the -SNAPSHOT suffix.
> Also the comment of Dan Kulp about the extra directory structure in
> the main file build/avro-1.9.0 directory would be a nice extra thing
> to fix.
>
> On Tue, Apr 30, 2019 at 12:54 PM Driesprong, Fokko <fo...@driesprong.frl> wrote:
> >
> > Hi everyone,
> >
> > Since the last release of Apache Avro 1.8.2 on May 31, 2017. Two years
> > later,
> > I'm thrilled to propose the following RC to be released as official Apache
> > Avro 1.9.0 release.
> >
> > The commit id is 24ff48c32d8d13433a1e31e485ef2af187d1eb62
> > * This corresponds to the tag: release-1.9.0-rc3
> > * https://github.com/apache/avro/releases/tag/release-1.9.0-rc3/
> >
> > The release tarball, signature, and checksums are here:
> > * https://dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc3/
> >
> > You can find the KEYS file here:
> > * https://dist.apache.org/repos/dist/dev/avro/KEYS
> >
> > Binary artifacts for Java are staged in Nexus here:
> > *
> > https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.0/
> >
> > This release includes 272 Jira issues:
> > https://issues.apache.org/jira/projects/AVRO/versions/12333394
> > * Deprecate Joda-Time in favor of Java8 JSR310 and setting it as default
> > * Remove support for Hadoop 1.x
> > * Move from Jackson 1.x to 2.9
> > * Add ZStandard Codec
> > * Lots of updates on the dependencies to fix CVE's
> > * and many, many more!
> >
> > Since RC1, two commits have been added:
> > * https://jira.apache.org/jira/browse/AVRO-2381
> > * https://jira.apache.org/jira/browse/AVRO-2383
> >
> > Since RC2 the SHA1/MD5 checksums have been replaced with SHA512
> >
> > Please download, verify, and test. This vote will remain open for at least
> > 72 hours. Given sufficient votes, I would like to close it on or about
> > midnight
> > on Saturday, 4th of May 2019.
> >
> > [ ] +1 Release this as Apache Avro 1.9.0
> > [ ] +0
> > [ ] -1 Do not release this because...
> >
> > Consider this a +1 (non-binding) from my side:
> > * Compiled the new version of Parquet against the Divolte collector and
> > Apache Parquet
> >
> > Cheers, Fokko Driesprong
Re: [VOTE] Release Apache Avro 1.9.0 RC3
Posted by Ismaël Mejía <ie...@gmail.com>.
Sorry, playing the killjoy again.
It seems the files (and more critical the poms) still have the -SNAPSHOT suffix.
Also the comment of Dan Kulp about the extra directory structure in
the main file build/avro-1.9.0 directory would be a nice extra thing
to fix.
On Tue, Apr 30, 2019 at 12:54 PM Driesprong, Fokko <fo...@driesprong.frl> wrote:
>
> Hi everyone,
>
> Since the last release of Apache Avro 1.8.2 on May 31, 2017. Two years
> later,
> I'm thrilled to propose the following RC to be released as official Apache
> Avro 1.9.0 release.
>
> The commit id is 24ff48c32d8d13433a1e31e485ef2af187d1eb62
> * This corresponds to the tag: release-1.9.0-rc3
> * https://github.com/apache/avro/releases/tag/release-1.9.0-rc3/
>
> The release tarball, signature, and checksums are here:
> * https://dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc3/
>
> You can find the KEYS file here:
> * https://dist.apache.org/repos/dist/dev/avro/KEYS
>
> Binary artifacts for Java are staged in Nexus here:
> *
> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.0/
>
> This release includes 272 Jira issues:
> https://issues.apache.org/jira/projects/AVRO/versions/12333394
> * Deprecate Joda-Time in favor of Java8 JSR310 and setting it as default
> * Remove support for Hadoop 1.x
> * Move from Jackson 1.x to 2.9
> * Add ZStandard Codec
> * Lots of updates on the dependencies to fix CVE's
> * and many, many more!
>
> Since RC1, two commits have been added:
> * https://jira.apache.org/jira/browse/AVRO-2381
> * https://jira.apache.org/jira/browse/AVRO-2383
>
> Since RC2 the SHA1/MD5 checksums have been replaced with SHA512
>
> Please download, verify, and test. This vote will remain open for at least
> 72 hours. Given sufficient votes, I would like to close it on or about
> midnight
> on Saturday, 4th of May 2019.
>
> [ ] +1 Release this as Apache Avro 1.9.0
> [ ] +0
> [ ] -1 Do not release this because...
>
> Consider this a +1 (non-binding) from my side:
> * Compiled the new version of Parquet against the Divolte collector and
> Apache Parquet
>
> Cheers, Fokko Driesprong
Re: [VOTE] Release Apache Avro 1.9.0 RC3
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Canceling the vote.
Thanks Karin for giving 1.9.0 RC a try. I'll release RC4 soon with your
patch.
Cheers, Fokko
Op vr 3 mei 2019 om 12:33 schreef Nandor Kollar
<nk...@cloudera.com.invalid>:
> Okay, I created a Jira: https://issues.apache.org/jira/browse/AVRO-2386
>
> Thanks Katrin, nice catch! You can open a PR for
> https://github.com/apache/avro.
>
> Because this is a regression, I vote -1 (non-binding) on RC3
>
> On Fri, May 3, 2019 at 12:20 PM Katrin Skoglund <Katrin.Skoglund@avanza.se
> >
> wrote:
>
> > I realized I don't have a Jira account, should I sign up for one?
> >
> > On 2019-05-03, 12:09, "Nandor Kollar" <nk...@cloudera.com.INVALID>
> > wrote:
> >
> > Ah, okay, I see. Well, in this case it is a release blocker then,
> > since it
> > is indeed a regression. If you've an account to Apache Jira, please
> > file a
> > Jira here: https://issues.apache.org/jira/projects/AVRO/issues
> >
> > On Fri, May 3, 2019 at 11:38 AM Katrin Skoglund <
> > Katrin.Skoglund@avanza.se>
> > wrote:
> >
> > > Hi Nandor!
> > >
> > > Ah, I wasn't aware that the method is new. What I actually meant
> was
> > that
> > > code generation that previously worked now generates code that no
> > longer
> > > compiles, which I would say is a regression. The implementation of
> > > customEncode() in the class representing the outer record calls the
> > > corresponding method on its nested records, which won't compile if
> > they are
> > > in different packages. This did not happen before since the method
> > did not
> > > exist, so the code compiled. Here's an example schema that
> > reproduces the
> > > problem:
> > >
> > > {"namespace": "org.apache.avro.codegentest.some",
> > > "type": "record",
> > > "name": "NestedSomeNamespaceRecord",
> > > "doc" : "Test nested types with different namespace than the
> outer
> > type",
> > > "fields": [
> > > {
> > > "name": "nestedRecord",
> > > "type": {
> > > "namespace": "org.apache.avro.codegentest.other",
> > > "type": "record",
> > > "name": "NestedOtherNamespaceRecord",
> > > "fields": [
> > > {
> > > "name": "someField",
> > > "type": "int"
> > > }
> > > ]
> > > }
> > > }]
> > > }
> > >
> > > And an example from the generated code:
> > >
> > > @Override protected void customEncode(org.apache.avro.io.Encoder
> > out)
> > > throws java.io.IOException
> > > {
> > > this.nestedRecord.customEncode(out);
> > >
> > > }
> > >
> > > //Katrin
> > >
> > >
> > > On 2019-05-03, 11:15, "Nandor Kollar" <nkollar@cloudera.com.INVALID
> >
> > > wrote:
> > >
> > > Hi Katrin,
> > >
> > > It appears to me, that these methods didn't exist in previous
> > Avro
> > > versions, they were added in
> > https://github.com/apache/avro/pull/350
> > > and
> > > were renamed and reduced their visibility in
> > >
> > >
> >
> https://github.com/apache/avro/pull/350/commits/d320b6b536c49b485be45286dbdb73226eef4b35
> > > .
> > > Actually it looks like the version with public visibility
> wasn't
> > even
> > > merged to master.
> > >
> > > Are you using a fork of Avro where these method were public, or
> > am I
> > > missing something? I'm not against of making these methods
> public
> > > (however
> > > I guess the author of the change had some reason to reduce the
> > scope
> > > form
> > > public to protected), however don't think this should be a
> > blocker for
> > > the
> > > release, as this doesn't seem to be a regression.
> > >
> > > Thanks,
> > > Nandor
> > >
> > > On Fri, May 3, 2019 at 10:43 AM Katrin Skoglund <
> > > Katrin.Skoglund@avanza.se>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I just managed to test the new RC, specifically the Java code
> > > generation,
> > > > with one of the libraries we use. I then noticed that their
> > > generated java
> > > > code no longer compiles because of a change in access level
> of
> > two
> > > methods.
> > > >
> > > > The generated methods customEncode and customDecode are
> > protected in
> > > this
> > > > version of Avro. They used to be public. This means that the
> > > generated java
> > > > code for schemas using nested records with different
> > namespaces will
> > > no
> > > > longer compile.
> > > >
> > > > I think it would be good to fix this before releasing since
> it
> > is a
> > > real
> > > > easy fix, unless there is a good reason why the access level
> > was
> > > changed to
> > > > protected?
> > > >
> > > > I'll start a PR for this anyway, please let me know if you
> > want the
> > > fix in
> > > > some other way or if I should create a Jira first (new to the
> > process
> > > > here).
> > > >
> > > > //Katrin
> > > >
> > > >
> > > > On 2019-04-30, 12:55, "Driesprong, Fokko"
> <fokko@driesprong.frl
> > >
> > > wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > Since the last release of Apache Avro 1.8.2 on May 31,
> > 2017. Two
> > > years
> > > > later,
> > > > I'm thrilled to propose the following RC to be released
> as
> > > official
> > > > Apache
> > > > Avro 1.9.0 release.
> > > >
> > > > The commit id is 24ff48c32d8d13433a1e31e485ef2af187d1eb62
> > > > * This corresponds to the tag: release-1.9.0-rc3
> > > > *
> > https://github.com/apache/avro/releases/tag/release-1.9.0-rc3/
> > > >
> > > > The release tarball, signature, and checksums are here:
> > > > *
> > https://dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc3/
> > > >
> > > > You can find the KEYS file here:
> > > > * https://dist.apache.org/repos/dist/dev/avro/KEYS
> > > >
> > > > Binary artifacts for Java are staged in Nexus here:
> > > > *
> > > >
> > > >
> > >
> >
> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.0/
> > > >
> > > > This release includes 272 Jira issues:
> > > >
> > https://issues.apache.org/jira/projects/AVRO/versions/12333394
> > > > * Deprecate Joda-Time in favor of Java8 JSR310 and
> setting
> > it as
> > > > default
> > > > * Remove support for Hadoop 1.x
> > > > * Move from Jackson 1.x to 2.9
> > > > * Add ZStandard Codec
> > > > * Lots of updates on the dependencies to fix CVE's
> > > > * and many, many more!
> > > >
> > > > Since RC1, two commits have been added:
> > > > * https://jira.apache.org/jira/browse/AVRO-2381
> > > > * https://jira.apache.org/jira/browse/AVRO-2383
> > > >
> > > > Since RC2 the SHA1/MD5 checksums have been replaced with
> > SHA512
> > > >
> > > > Please download, verify, and test. This vote will remain
> > open
> > > for at
> > > > least
> > > > 72 hours. Given sufficient votes, I would like to close
> it
> > on or
> > > about
> > > > midnight
> > > > on Saturday, 4th of May 2019.
> > > >
> > > > [ ] +1 Release this as Apache Avro 1.9.0
> > > > [ ] +0
> > > > [ ] -1 Do not release this because...
> > > >
> > > > Consider this a +1 (non-binding) from my side:
> > > > * Compiled the new version of Parquet against the Divolte
> > > collector and
> > > > Apache Parquet
> > > >
> > > > Cheers, Fokko Driesprong
> > > >
> > > >
> > > >
> > >
> > >
> > >
> >
> >
> >
>
Re: [VOTE] Release Apache Avro 1.9.0 RC3
Posted by Nandor Kollar <nk...@cloudera.com.INVALID>.
Okay, I created a Jira: https://issues.apache.org/jira/browse/AVRO-2386
Thanks Katrin, nice catch! You can open a PR for
https://github.com/apache/avro.
Because this is a regression, I vote -1 (non-binding) on RC3
On Fri, May 3, 2019 at 12:20 PM Katrin Skoglund <Ka...@avanza.se>
wrote:
> I realized I don't have a Jira account, should I sign up for one?
>
> On 2019-05-03, 12:09, "Nandor Kollar" <nk...@cloudera.com.INVALID>
> wrote:
>
> Ah, okay, I see. Well, in this case it is a release blocker then,
> since it
> is indeed a regression. If you've an account to Apache Jira, please
> file a
> Jira here: https://issues.apache.org/jira/projects/AVRO/issues
>
> On Fri, May 3, 2019 at 11:38 AM Katrin Skoglund <
> Katrin.Skoglund@avanza.se>
> wrote:
>
> > Hi Nandor!
> >
> > Ah, I wasn't aware that the method is new. What I actually meant was
> that
> > code generation that previously worked now generates code that no
> longer
> > compiles, which I would say is a regression. The implementation of
> > customEncode() in the class representing the outer record calls the
> > corresponding method on its nested records, which won't compile if
> they are
> > in different packages. This did not happen before since the method
> did not
> > exist, so the code compiled. Here's an example schema that
> reproduces the
> > problem:
> >
> > {"namespace": "org.apache.avro.codegentest.some",
> > "type": "record",
> > "name": "NestedSomeNamespaceRecord",
> > "doc" : "Test nested types with different namespace than the outer
> type",
> > "fields": [
> > {
> > "name": "nestedRecord",
> > "type": {
> > "namespace": "org.apache.avro.codegentest.other",
> > "type": "record",
> > "name": "NestedOtherNamespaceRecord",
> > "fields": [
> > {
> > "name": "someField",
> > "type": "int"
> > }
> > ]
> > }
> > }]
> > }
> >
> > And an example from the generated code:
> >
> > @Override protected void customEncode(org.apache.avro.io.Encoder
> out)
> > throws java.io.IOException
> > {
> > this.nestedRecord.customEncode(out);
> >
> > }
> >
> > //Katrin
> >
> >
> > On 2019-05-03, 11:15, "Nandor Kollar" <nk...@cloudera.com.INVALID>
> > wrote:
> >
> > Hi Katrin,
> >
> > It appears to me, that these methods didn't exist in previous
> Avro
> > versions, they were added in
> https://github.com/apache/avro/pull/350
> > and
> > were renamed and reduced their visibility in
> >
> >
> https://github.com/apache/avro/pull/350/commits/d320b6b536c49b485be45286dbdb73226eef4b35
> > .
> > Actually it looks like the version with public visibility wasn't
> even
> > merged to master.
> >
> > Are you using a fork of Avro where these method were public, or
> am I
> > missing something? I'm not against of making these methods public
> > (however
> > I guess the author of the change had some reason to reduce the
> scope
> > form
> > public to protected), however don't think this should be a
> blocker for
> > the
> > release, as this doesn't seem to be a regression.
> >
> > Thanks,
> > Nandor
> >
> > On Fri, May 3, 2019 at 10:43 AM Katrin Skoglund <
> > Katrin.Skoglund@avanza.se>
> > wrote:
> >
> > > Hi all,
> > >
> > > I just managed to test the new RC, specifically the Java code
> > generation,
> > > with one of the libraries we use. I then noticed that their
> > generated java
> > > code no longer compiles because of a change in access level of
> two
> > methods.
> > >
> > > The generated methods customEncode and customDecode are
> protected in
> > this
> > > version of Avro. They used to be public. This means that the
> > generated java
> > > code for schemas using nested records with different
> namespaces will
> > no
> > > longer compile.
> > >
> > > I think it would be good to fix this before releasing since it
> is a
> > real
> > > easy fix, unless there is a good reason why the access level
> was
> > changed to
> > > protected?
> > >
> > > I'll start a PR for this anyway, please let me know if you
> want the
> > fix in
> > > some other way or if I should create a Jira first (new to the
> process
> > > here).
> > >
> > > //Katrin
> > >
> > >
> > > On 2019-04-30, 12:55, "Driesprong, Fokko" <fokko@driesprong.frl
> >
> > wrote:
> > >
> > > Hi everyone,
> > >
> > > Since the last release of Apache Avro 1.8.2 on May 31,
> 2017. Two
> > years
> > > later,
> > > I'm thrilled to propose the following RC to be released as
> > official
> > > Apache
> > > Avro 1.9.0 release.
> > >
> > > The commit id is 24ff48c32d8d13433a1e31e485ef2af187d1eb62
> > > * This corresponds to the tag: release-1.9.0-rc3
> > > *
> https://github.com/apache/avro/releases/tag/release-1.9.0-rc3/
> > >
> > > The release tarball, signature, and checksums are here:
> > > *
> https://dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc3/
> > >
> > > You can find the KEYS file here:
> > > * https://dist.apache.org/repos/dist/dev/avro/KEYS
> > >
> > > Binary artifacts for Java are staged in Nexus here:
> > > *
> > >
> > >
> >
> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.0/
> > >
> > > This release includes 272 Jira issues:
> > >
> https://issues.apache.org/jira/projects/AVRO/versions/12333394
> > > * Deprecate Joda-Time in favor of Java8 JSR310 and setting
> it as
> > > default
> > > * Remove support for Hadoop 1.x
> > > * Move from Jackson 1.x to 2.9
> > > * Add ZStandard Codec
> > > * Lots of updates on the dependencies to fix CVE's
> > > * and many, many more!
> > >
> > > Since RC1, two commits have been added:
> > > * https://jira.apache.org/jira/browse/AVRO-2381
> > > * https://jira.apache.org/jira/browse/AVRO-2383
> > >
> > > Since RC2 the SHA1/MD5 checksums have been replaced with
> SHA512
> > >
> > > Please download, verify, and test. This vote will remain
> open
> > for at
> > > least
> > > 72 hours. Given sufficient votes, I would like to close it
> on or
> > about
> > > midnight
> > > on Saturday, 4th of May 2019.
> > >
> > > [ ] +1 Release this as Apache Avro 1.9.0
> > > [ ] +0
> > > [ ] -1 Do not release this because...
> > >
> > > Consider this a +1 (non-binding) from my side:
> > > * Compiled the new version of Parquet against the Divolte
> > collector and
> > > Apache Parquet
> > >
> > > Cheers, Fokko Driesprong
> > >
> > >
> > >
> >
> >
> >
>
>
>
Re: [VOTE] Release Apache Avro 1.9.0 RC3
Posted by Katrin Skoglund <Ka...@avanza.se>.
I realized I don't have a Jira account, should I sign up for one?
On 2019-05-03, 12:09, "Nandor Kollar" <nk...@cloudera.com.INVALID> wrote:
Ah, okay, I see. Well, in this case it is a release blocker then, since it
is indeed a regression. If you've an account to Apache Jira, please file a
Jira here: https://issues.apache.org/jira/projects/AVRO/issues
On Fri, May 3, 2019 at 11:38 AM Katrin Skoglund <Ka...@avanza.se>
wrote:
> Hi Nandor!
>
> Ah, I wasn't aware that the method is new. What I actually meant was that
> code generation that previously worked now generates code that no longer
> compiles, which I would say is a regression. The implementation of
> customEncode() in the class representing the outer record calls the
> corresponding method on its nested records, which won't compile if they are
> in different packages. This did not happen before since the method did not
> exist, so the code compiled. Here's an example schema that reproduces the
> problem:
>
> {"namespace": "org.apache.avro.codegentest.some",
> "type": "record",
> "name": "NestedSomeNamespaceRecord",
> "doc" : "Test nested types with different namespace than the outer type",
> "fields": [
> {
> "name": "nestedRecord",
> "type": {
> "namespace": "org.apache.avro.codegentest.other",
> "type": "record",
> "name": "NestedOtherNamespaceRecord",
> "fields": [
> {
> "name": "someField",
> "type": "int"
> }
> ]
> }
> }]
> }
>
> And an example from the generated code:
>
> @Override protected void customEncode(org.apache.avro.io.Encoder out)
> throws java.io.IOException
> {
> this.nestedRecord.customEncode(out);
>
> }
>
> //Katrin
>
>
> On 2019-05-03, 11:15, "Nandor Kollar" <nk...@cloudera.com.INVALID>
> wrote:
>
> Hi Katrin,
>
> It appears to me, that these methods didn't exist in previous Avro
> versions, they were added in https://github.com/apache/avro/pull/350
> and
> were renamed and reduced their visibility in
>
> https://github.com/apache/avro/pull/350/commits/d320b6b536c49b485be45286dbdb73226eef4b35
> .
> Actually it looks like the version with public visibility wasn't even
> merged to master.
>
> Are you using a fork of Avro where these method were public, or am I
> missing something? I'm not against of making these methods public
> (however
> I guess the author of the change had some reason to reduce the scope
> form
> public to protected), however don't think this should be a blocker for
> the
> release, as this doesn't seem to be a regression.
>
> Thanks,
> Nandor
>
> On Fri, May 3, 2019 at 10:43 AM Katrin Skoglund <
> Katrin.Skoglund@avanza.se>
> wrote:
>
> > Hi all,
> >
> > I just managed to test the new RC, specifically the Java code
> generation,
> > with one of the libraries we use. I then noticed that their
> generated java
> > code no longer compiles because of a change in access level of two
> methods.
> >
> > The generated methods customEncode and customDecode are protected in
> this
> > version of Avro. They used to be public. This means that the
> generated java
> > code for schemas using nested records with different namespaces will
> no
> > longer compile.
> >
> > I think it would be good to fix this before releasing since it is a
> real
> > easy fix, unless there is a good reason why the access level was
> changed to
> > protected?
> >
> > I'll start a PR for this anyway, please let me know if you want the
> fix in
> > some other way or if I should create a Jira first (new to the process
> > here).
> >
> > //Katrin
> >
> >
> > On 2019-04-30, 12:55, "Driesprong, Fokko" <fo...@driesprong.frl>
> wrote:
> >
> > Hi everyone,
> >
> > Since the last release of Apache Avro 1.8.2 on May 31, 2017. Two
> years
> > later,
> > I'm thrilled to propose the following RC to be released as
> official
> > Apache
> > Avro 1.9.0 release.
> >
> > The commit id is 24ff48c32d8d13433a1e31e485ef2af187d1eb62
> > * This corresponds to the tag: release-1.9.0-rc3
> > * https://github.com/apache/avro/releases/tag/release-1.9.0-rc3/
> >
> > The release tarball, signature, and checksums are here:
> > * https://dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc3/
> >
> > You can find the KEYS file here:
> > * https://dist.apache.org/repos/dist/dev/avro/KEYS
> >
> > Binary artifacts for Java are staged in Nexus here:
> > *
> >
> >
> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.0/
> >
> > This release includes 272 Jira issues:
> > https://issues.apache.org/jira/projects/AVRO/versions/12333394
> > * Deprecate Joda-Time in favor of Java8 JSR310 and setting it as
> > default
> > * Remove support for Hadoop 1.x
> > * Move from Jackson 1.x to 2.9
> > * Add ZStandard Codec
> > * Lots of updates on the dependencies to fix CVE's
> > * and many, many more!
> >
> > Since RC1, two commits have been added:
> > * https://jira.apache.org/jira/browse/AVRO-2381
> > * https://jira.apache.org/jira/browse/AVRO-2383
> >
> > Since RC2 the SHA1/MD5 checksums have been replaced with SHA512
> >
> > Please download, verify, and test. This vote will remain open
> for at
> > least
> > 72 hours. Given sufficient votes, I would like to close it on or
> about
> > midnight
> > on Saturday, 4th of May 2019.
> >
> > [ ] +1 Release this as Apache Avro 1.9.0
> > [ ] +0
> > [ ] -1 Do not release this because...
> >
> > Consider this a +1 (non-binding) from my side:
> > * Compiled the new version of Parquet against the Divolte
> collector and
> > Apache Parquet
> >
> > Cheers, Fokko Driesprong
> >
> >
> >
>
>
>
Re: [VOTE] Release Apache Avro 1.9.0 RC3
Posted by Nandor Kollar <nk...@cloudera.com.INVALID>.
Ah, okay, I see. Well, in this case it is a release blocker then, since it
is indeed a regression. If you've an account to Apache Jira, please file a
Jira here: https://issues.apache.org/jira/projects/AVRO/issues
On Fri, May 3, 2019 at 11:38 AM Katrin Skoglund <Ka...@avanza.se>
wrote:
> Hi Nandor!
>
> Ah, I wasn't aware that the method is new. What I actually meant was that
> code generation that previously worked now generates code that no longer
> compiles, which I would say is a regression. The implementation of
> customEncode() in the class representing the outer record calls the
> corresponding method on its nested records, which won't compile if they are
> in different packages. This did not happen before since the method did not
> exist, so the code compiled. Here's an example schema that reproduces the
> problem:
>
> {"namespace": "org.apache.avro.codegentest.some",
> "type": "record",
> "name": "NestedSomeNamespaceRecord",
> "doc" : "Test nested types with different namespace than the outer type",
> "fields": [
> {
> "name": "nestedRecord",
> "type": {
> "namespace": "org.apache.avro.codegentest.other",
> "type": "record",
> "name": "NestedOtherNamespaceRecord",
> "fields": [
> {
> "name": "someField",
> "type": "int"
> }
> ]
> }
> }]
> }
>
> And an example from the generated code:
>
> @Override protected void customEncode(org.apache.avro.io.Encoder out)
> throws java.io.IOException
> {
> this.nestedRecord.customEncode(out);
>
> }
>
> //Katrin
>
>
> On 2019-05-03, 11:15, "Nandor Kollar" <nk...@cloudera.com.INVALID>
> wrote:
>
> Hi Katrin,
>
> It appears to me, that these methods didn't exist in previous Avro
> versions, they were added in https://github.com/apache/avro/pull/350
> and
> were renamed and reduced their visibility in
>
> https://github.com/apache/avro/pull/350/commits/d320b6b536c49b485be45286dbdb73226eef4b35
> .
> Actually it looks like the version with public visibility wasn't even
> merged to master.
>
> Are you using a fork of Avro where these method were public, or am I
> missing something? I'm not against of making these methods public
> (however
> I guess the author of the change had some reason to reduce the scope
> form
> public to protected), however don't think this should be a blocker for
> the
> release, as this doesn't seem to be a regression.
>
> Thanks,
> Nandor
>
> On Fri, May 3, 2019 at 10:43 AM Katrin Skoglund <
> Katrin.Skoglund@avanza.se>
> wrote:
>
> > Hi all,
> >
> > I just managed to test the new RC, specifically the Java code
> generation,
> > with one of the libraries we use. I then noticed that their
> generated java
> > code no longer compiles because of a change in access level of two
> methods.
> >
> > The generated methods customEncode and customDecode are protected in
> this
> > version of Avro. They used to be public. This means that the
> generated java
> > code for schemas using nested records with different namespaces will
> no
> > longer compile.
> >
> > I think it would be good to fix this before releasing since it is a
> real
> > easy fix, unless there is a good reason why the access level was
> changed to
> > protected?
> >
> > I'll start a PR for this anyway, please let me know if you want the
> fix in
> > some other way or if I should create a Jira first (new to the process
> > here).
> >
> > //Katrin
> >
> >
> > On 2019-04-30, 12:55, "Driesprong, Fokko" <fo...@driesprong.frl>
> wrote:
> >
> > Hi everyone,
> >
> > Since the last release of Apache Avro 1.8.2 on May 31, 2017. Two
> years
> > later,
> > I'm thrilled to propose the following RC to be released as
> official
> > Apache
> > Avro 1.9.0 release.
> >
> > The commit id is 24ff48c32d8d13433a1e31e485ef2af187d1eb62
> > * This corresponds to the tag: release-1.9.0-rc3
> > * https://github.com/apache/avro/releases/tag/release-1.9.0-rc3/
> >
> > The release tarball, signature, and checksums are here:
> > * https://dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc3/
> >
> > You can find the KEYS file here:
> > * https://dist.apache.org/repos/dist/dev/avro/KEYS
> >
> > Binary artifacts for Java are staged in Nexus here:
> > *
> >
> >
> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.0/
> >
> > This release includes 272 Jira issues:
> > https://issues.apache.org/jira/projects/AVRO/versions/12333394
> > * Deprecate Joda-Time in favor of Java8 JSR310 and setting it as
> > default
> > * Remove support for Hadoop 1.x
> > * Move from Jackson 1.x to 2.9
> > * Add ZStandard Codec
> > * Lots of updates on the dependencies to fix CVE's
> > * and many, many more!
> >
> > Since RC1, two commits have been added:
> > * https://jira.apache.org/jira/browse/AVRO-2381
> > * https://jira.apache.org/jira/browse/AVRO-2383
> >
> > Since RC2 the SHA1/MD5 checksums have been replaced with SHA512
> >
> > Please download, verify, and test. This vote will remain open
> for at
> > least
> > 72 hours. Given sufficient votes, I would like to close it on or
> about
> > midnight
> > on Saturday, 4th of May 2019.
> >
> > [ ] +1 Release this as Apache Avro 1.9.0
> > [ ] +0
> > [ ] -1 Do not release this because...
> >
> > Consider this a +1 (non-binding) from my side:
> > * Compiled the new version of Parquet against the Divolte
> collector and
> > Apache Parquet
> >
> > Cheers, Fokko Driesprong
> >
> >
> >
>
>
>
Re: [VOTE] Release Apache Avro 1.9.0 RC3
Posted by Katrin Skoglund <Ka...@avanza.se>.
Hi Nandor!
Ah, I wasn't aware that the method is new. What I actually meant was that code generation that previously worked now generates code that no longer compiles, which I would say is a regression. The implementation of customEncode() in the class representing the outer record calls the corresponding method on its nested records, which won't compile if they are in different packages. This did not happen before since the method did not exist, so the code compiled. Here's an example schema that reproduces the problem:
{"namespace": "org.apache.avro.codegentest.some",
"type": "record",
"name": "NestedSomeNamespaceRecord",
"doc" : "Test nested types with different namespace than the outer type",
"fields": [
{
"name": "nestedRecord",
"type": {
"namespace": "org.apache.avro.codegentest.other",
"type": "record",
"name": "NestedOtherNamespaceRecord",
"fields": [
{
"name": "someField",
"type": "int"
}
]
}
}]
}
And an example from the generated code:
@Override protected void customEncode(org.apache.avro.io.Encoder out)
throws java.io.IOException
{
this.nestedRecord.customEncode(out);
}
//Katrin
On 2019-05-03, 11:15, "Nandor Kollar" <nk...@cloudera.com.INVALID> wrote:
Hi Katrin,
It appears to me, that these methods didn't exist in previous Avro
versions, they were added in https://github.com/apache/avro/pull/350 and
were renamed and reduced their visibility in
https://github.com/apache/avro/pull/350/commits/d320b6b536c49b485be45286dbdb73226eef4b35.
Actually it looks like the version with public visibility wasn't even
merged to master.
Are you using a fork of Avro where these method were public, or am I
missing something? I'm not against of making these methods public (however
I guess the author of the change had some reason to reduce the scope form
public to protected), however don't think this should be a blocker for the
release, as this doesn't seem to be a regression.
Thanks,
Nandor
On Fri, May 3, 2019 at 10:43 AM Katrin Skoglund <Ka...@avanza.se>
wrote:
> Hi all,
>
> I just managed to test the new RC, specifically the Java code generation,
> with one of the libraries we use. I then noticed that their generated java
> code no longer compiles because of a change in access level of two methods.
>
> The generated methods customEncode and customDecode are protected in this
> version of Avro. They used to be public. This means that the generated java
> code for schemas using nested records with different namespaces will no
> longer compile.
>
> I think it would be good to fix this before releasing since it is a real
> easy fix, unless there is a good reason why the access level was changed to
> protected?
>
> I'll start a PR for this anyway, please let me know if you want the fix in
> some other way or if I should create a Jira first (new to the process
> here).
>
> //Katrin
>
>
> On 2019-04-30, 12:55, "Driesprong, Fokko" <fo...@driesprong.frl> wrote:
>
> Hi everyone,
>
> Since the last release of Apache Avro 1.8.2 on May 31, 2017. Two years
> later,
> I'm thrilled to propose the following RC to be released as official
> Apache
> Avro 1.9.0 release.
>
> The commit id is 24ff48c32d8d13433a1e31e485ef2af187d1eb62
> * This corresponds to the tag: release-1.9.0-rc3
> * https://github.com/apache/avro/releases/tag/release-1.9.0-rc3/
>
> The release tarball, signature, and checksums are here:
> * https://dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc3/
>
> You can find the KEYS file here:
> * https://dist.apache.org/repos/dist/dev/avro/KEYS
>
> Binary artifacts for Java are staged in Nexus here:
> *
>
> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.0/
>
> This release includes 272 Jira issues:
> https://issues.apache.org/jira/projects/AVRO/versions/12333394
> * Deprecate Joda-Time in favor of Java8 JSR310 and setting it as
> default
> * Remove support for Hadoop 1.x
> * Move from Jackson 1.x to 2.9
> * Add ZStandard Codec
> * Lots of updates on the dependencies to fix CVE's
> * and many, many more!
>
> Since RC1, two commits have been added:
> * https://jira.apache.org/jira/browse/AVRO-2381
> * https://jira.apache.org/jira/browse/AVRO-2383
>
> Since RC2 the SHA1/MD5 checksums have been replaced with SHA512
>
> Please download, verify, and test. This vote will remain open for at
> least
> 72 hours. Given sufficient votes, I would like to close it on or about
> midnight
> on Saturday, 4th of May 2019.
>
> [ ] +1 Release this as Apache Avro 1.9.0
> [ ] +0
> [ ] -1 Do not release this because...
>
> Consider this a +1 (non-binding) from my side:
> * Compiled the new version of Parquet against the Divolte collector and
> Apache Parquet
>
> Cheers, Fokko Driesprong
>
>
>
Re: [VOTE] Release Apache Avro 1.9.0 RC3
Posted by Nandor Kollar <nk...@cloudera.com.INVALID>.
Hi Katrin,
It appears to me, that these methods didn't exist in previous Avro
versions, they were added in https://github.com/apache/avro/pull/350 and
were renamed and reduced their visibility in
https://github.com/apache/avro/pull/350/commits/d320b6b536c49b485be45286dbdb73226eef4b35.
Actually it looks like the version with public visibility wasn't even
merged to master.
Are you using a fork of Avro where these method were public, or am I
missing something? I'm not against of making these methods public (however
I guess the author of the change had some reason to reduce the scope form
public to protected), however don't think this should be a blocker for the
release, as this doesn't seem to be a regression.
Thanks,
Nandor
On Fri, May 3, 2019 at 10:43 AM Katrin Skoglund <Ka...@avanza.se>
wrote:
> Hi all,
>
> I just managed to test the new RC, specifically the Java code generation,
> with one of the libraries we use. I then noticed that their generated java
> code no longer compiles because of a change in access level of two methods.
>
> The generated methods customEncode and customDecode are protected in this
> version of Avro. They used to be public. This means that the generated java
> code for schemas using nested records with different namespaces will no
> longer compile.
>
> I think it would be good to fix this before releasing since it is a real
> easy fix, unless there is a good reason why the access level was changed to
> protected?
>
> I'll start a PR for this anyway, please let me know if you want the fix in
> some other way or if I should create a Jira first (new to the process
> here).
>
> //Katrin
>
>
> On 2019-04-30, 12:55, "Driesprong, Fokko" <fo...@driesprong.frl> wrote:
>
> Hi everyone,
>
> Since the last release of Apache Avro 1.8.2 on May 31, 2017. Two years
> later,
> I'm thrilled to propose the following RC to be released as official
> Apache
> Avro 1.9.0 release.
>
> The commit id is 24ff48c32d8d13433a1e31e485ef2af187d1eb62
> * This corresponds to the tag: release-1.9.0-rc3
> * https://github.com/apache/avro/releases/tag/release-1.9.0-rc3/
>
> The release tarball, signature, and checksums are here:
> * https://dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc3/
>
> You can find the KEYS file here:
> * https://dist.apache.org/repos/dist/dev/avro/KEYS
>
> Binary artifacts for Java are staged in Nexus here:
> *
>
> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.0/
>
> This release includes 272 Jira issues:
> https://issues.apache.org/jira/projects/AVRO/versions/12333394
> * Deprecate Joda-Time in favor of Java8 JSR310 and setting it as
> default
> * Remove support for Hadoop 1.x
> * Move from Jackson 1.x to 2.9
> * Add ZStandard Codec
> * Lots of updates on the dependencies to fix CVE's
> * and many, many more!
>
> Since RC1, two commits have been added:
> * https://jira.apache.org/jira/browse/AVRO-2381
> * https://jira.apache.org/jira/browse/AVRO-2383
>
> Since RC2 the SHA1/MD5 checksums have been replaced with SHA512
>
> Please download, verify, and test. This vote will remain open for at
> least
> 72 hours. Given sufficient votes, I would like to close it on or about
> midnight
> on Saturday, 4th of May 2019.
>
> [ ] +1 Release this as Apache Avro 1.9.0
> [ ] +0
> [ ] -1 Do not release this because...
>
> Consider this a +1 (non-binding) from my side:
> * Compiled the new version of Parquet against the Divolte collector and
> Apache Parquet
>
> Cheers, Fokko Driesprong
>
>
>
Re: [VOTE] Release Apache Avro 1.9.0 RC3
Posted by Katrin Skoglund <Ka...@avanza.se>.
Hi all,
I just managed to test the new RC, specifically the Java code generation, with one of the libraries we use. I then noticed that their generated java code no longer compiles because of a change in access level of two methods.
The generated methods customEncode and customDecode are protected in this version of Avro. They used to be public. This means that the generated java code for schemas using nested records with different namespaces will no longer compile.
I think it would be good to fix this before releasing since it is a real easy fix, unless there is a good reason why the access level was changed to protected?
I'll start a PR for this anyway, please let me know if you want the fix in some other way or if I should create a Jira first (new to the process here).
//Katrin
On 2019-04-30, 12:55, "Driesprong, Fokko" <fo...@driesprong.frl> wrote:
Hi everyone,
Since the last release of Apache Avro 1.8.2 on May 31, 2017. Two years
later,
I'm thrilled to propose the following RC to be released as official Apache
Avro 1.9.0 release.
The commit id is 24ff48c32d8d13433a1e31e485ef2af187d1eb62
* This corresponds to the tag: release-1.9.0-rc3
* https://github.com/apache/avro/releases/tag/release-1.9.0-rc3/
The release tarball, signature, and checksums are here:
* https://dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc3/
You can find the KEYS file here:
* https://dist.apache.org/repos/dist/dev/avro/KEYS
Binary artifacts for Java are staged in Nexus here:
*
https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.0/
This release includes 272 Jira issues:
https://issues.apache.org/jira/projects/AVRO/versions/12333394
* Deprecate Joda-Time in favor of Java8 JSR310 and setting it as default
* Remove support for Hadoop 1.x
* Move from Jackson 1.x to 2.9
* Add ZStandard Codec
* Lots of updates on the dependencies to fix CVE's
* and many, many more!
Since RC1, two commits have been added:
* https://jira.apache.org/jira/browse/AVRO-2381
* https://jira.apache.org/jira/browse/AVRO-2383
Since RC2 the SHA1/MD5 checksums have been replaced with SHA512
Please download, verify, and test. This vote will remain open for at least
72 hours. Given sufficient votes, I would like to close it on or about
midnight
on Saturday, 4th of May 2019.
[ ] +1 Release this as Apache Avro 1.9.0
[ ] +0
[ ] -1 Do not release this because...
Consider this a +1 (non-binding) from my side:
* Compiled the new version of Parquet against the Divolte collector and
Apache Parquet
Cheers, Fokko Driesprong