You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by Markus Weimer <ma...@weimo.de> on 2016/09/08 22:21:14 UTC
[Discuss] Drop Avro in favor of Protobuf 3.x
Hi,
protocol buffers recently added .NET support, and Microsoft's Avro
implementation doesn't seem to go to .NET Core. This is an impediment
to be on CoreCLR on Linux. Hence, moving all serialization in REEF to
that new version of protobuf. We have discussed this (or rather:
hinted at this) in several threads, but I'd like to make this
discussion and subsequent decision more prominent.
Are there any concerns with this move? We'd remove the use of Avro and
move to protbuf 3.x for all use in REEF. For Hadoop and YARN, we'll
have to stick to protobuf 2.5, potentially incurring some shading.
Markus
Re: [Discuss] Drop Avro in favor of Protobuf 3.x
Posted by Byung-Gon Chun <bg...@gmail.com>.
Sounds great.
On Wed, Sep 14, 2016 at 7:53 PM, Douglas Service <ds...@gmail.com> wrote:
> Yes I agree, we should do a release just before dropping support for .NET
> 4.5.
>
> Since, no one has objected to replacing Avro with protobuf 3.x, I am going
> to start on that work. First I will spend some time coming to speed on the
> code and understanding the necessary changes, then I will start making the
> changes.
>
> Doug
>
> On Fri, Sep 9, 2016 at 10:23 AM, Markus Weimer <ma...@weimo.de> wrote:
>
> > On Fri, Sep 9, 2016 at 7:00 AM, Boris Shulman <sh...@gmail.com>
> wrote:
> > > As long as we can compile to .NET 4.5 we should be good.
> >
> > For the CoreCLR work, we will have to eventually drop support for .NET
> > 4.5. I suggest we make one last release of REEF right before doing
> > that. Sounds good?
> >
> > Markus
> >
>
--
Byung-Gon Chun
Re: [Discuss] Drop Avro in favor of Protobuf 3.x
Posted by Byung-Gon Chun <bg...@gmail.com>.
Thanks!
On Sun, Sep 18, 2016 at 5:31 PM, Markus Weimer <ma...@weimo.de> wrote:
> Doug filed [REEF-1566] as the umbrella JIRA for this work stream.
>
> Markus
>
> On Thu, Sep 15, 2016 at 10:38 AM, Mariia Mykhailova
> <ma...@microsoft.com> wrote:
> > For .NET code, we have tests in O.A.R.Tests.Functional which are
> supposed to run on YARN as well, but we've never executed them so there is
> still work to be done to run them and (more importantly) to verify the
> results of their execution. https://issues.apache.org/
> jira/browse/REEF-1541 tracks this work
> >
> > -Mariia
> >
> > -----Original Message-----
> > From: Markus Weimer [mailto:markus@weimo.de]
> > Sent: Thursday, September 15, 2016 9:11 AM
> > To: REEF Developers Mailinglist <de...@reef.apache.org>
> > Subject: Re: [Discuss] Drop Avro in favor of Protobuf 3.x
> >
> > On Thu, Sep 15, 2016 at 8:10 AM, Douglas Service <ds...@gmail.com>
> wrote:
> >> I will look to see if we have tests that verify communication with Yarn.
> >
> > The module reff-tests in the java code base contains many integration
> tests that can be run on a YARN cluster.
> >
> > Markus
>
--
Byung-Gon Chun
Re: [Discuss] Drop Avro in favor of Protobuf 3.x
Posted by Markus Weimer <ma...@weimo.de>.
Doug filed [REEF-1566] as the umbrella JIRA for this work stream.
Markus
On Thu, Sep 15, 2016 at 10:38 AM, Mariia Mykhailova
<ma...@microsoft.com> wrote:
> For .NET code, we have tests in O.A.R.Tests.Functional which are supposed to run on YARN as well, but we've never executed them so there is still work to be done to run them and (more importantly) to verify the results of their execution. https://issues.apache.org/jira/browse/REEF-1541 tracks this work
>
> -Mariia
>
> -----Original Message-----
> From: Markus Weimer [mailto:markus@weimo.de]
> Sent: Thursday, September 15, 2016 9:11 AM
> To: REEF Developers Mailinglist <de...@reef.apache.org>
> Subject: Re: [Discuss] Drop Avro in favor of Protobuf 3.x
>
> On Thu, Sep 15, 2016 at 8:10 AM, Douglas Service <ds...@gmail.com> wrote:
>> I will look to see if we have tests that verify communication with Yarn.
>
> The module reff-tests in the java code base contains many integration tests that can be run on a YARN cluster.
>
> Markus
RE: [Discuss] Drop Avro in favor of Protobuf 3.x
Posted by Mariia Mykhailova <ma...@microsoft.com>.
For .NET code, we have tests in O.A.R.Tests.Functional which are supposed to run on YARN as well, but we've never executed them so there is still work to be done to run them and (more importantly) to verify the results of their execution. https://issues.apache.org/jira/browse/REEF-1541 tracks this work
-Mariia
-----Original Message-----
From: Markus Weimer [mailto:markus@weimo.de]
Sent: Thursday, September 15, 2016 9:11 AM
To: REEF Developers Mailinglist <de...@reef.apache.org>
Subject: Re: [Discuss] Drop Avro in favor of Protobuf 3.x
On Thu, Sep 15, 2016 at 8:10 AM, Douglas Service <ds...@gmail.com> wrote:
> I will look to see if we have tests that verify communication with Yarn.
The module reff-tests in the java code base contains many integration tests that can be run on a YARN cluster.
Markus
Re: [Discuss] Drop Avro in favor of Protobuf 3.x
Posted by Markus Weimer <ma...@weimo.de>.
On Thu, Sep 15, 2016 at 8:10 AM, Douglas Service <ds...@gmail.com> wrote:
> I will look to see if we have tests that verify communication with Yarn.
The module reff-tests in the java code base contains many integration
tests that can be run on a YARN cluster.
Markus
Re: [Discuss] Drop Avro in favor of Protobuf 3.x
Posted by Douglas Service <ds...@gmail.com>.
Hello Jaliya,
Yes, we will use shading as previously discussed so we can continue to use
protobuf 2.5 to communicate with Yarn and 3.x to replace Avro. I will look
to see if we have tests that verify communication with Yarn. If not we will
need to write some.
Doug
On Thu, Sep 15, 2016 at 12:53 AM, Jaliya Ekanayake <ja...@microsoft.com>
wrote:
> I assume this will includes tasks to make sure the compatibility with Yarn
> which runs a previous version of protobuf?
>
> Thank you
> jaliya
>
> -----Original Message-----
> From: Markus Weimer [mailto:markus@weimo.de]
> Sent: Wednesday, September 14, 2016 8:38 PM
> To: REEF Developers Mailinglist <de...@reef.apache.org>
> Subject: Re: [Discuss] Drop Avro in favor of Protobuf 3.x
>
> On Wed, Sep 14, 2016 at 7:53 PM, Douglas Service <ds...@gmail.com>
> wrote:
> > Since, no one has objected to replacing Avro with protobuf 3.x, I am
> > going to start on that work. First I will spend some time coming to
> > speed on the code and understanding the necessary changes, then I will
> > start making the changes.
>
>
> +1
>
> Also, it would be great to have some breakdown of work in JIRA for this.
> We could, for example, move through the dependency chain.
>
> Markus
>
RE: [Discuss] Drop Avro in favor of Protobuf 3.x
Posted by Jaliya Ekanayake <ja...@microsoft.com>.
I assume this will includes tasks to make sure the compatibility with Yarn which runs a previous version of protobuf?
Thank you
jaliya
-----Original Message-----
From: Markus Weimer [mailto:markus@weimo.de]
Sent: Wednesday, September 14, 2016 8:38 PM
To: REEF Developers Mailinglist <de...@reef.apache.org>
Subject: Re: [Discuss] Drop Avro in favor of Protobuf 3.x
On Wed, Sep 14, 2016 at 7:53 PM, Douglas Service <ds...@gmail.com> wrote:
> Since, no one has objected to replacing Avro with protobuf 3.x, I am
> going to start on that work. First I will spend some time coming to
> speed on the code and understanding the necessary changes, then I will
> start making the changes.
+1
Also, it would be great to have some breakdown of work in JIRA for this. We could, for example, move through the dependency chain.
Markus
Re: [Discuss] Drop Avro in favor of Protobuf 3.x
Posted by Douglas Service <ds...@gmail.com>.
Yes, I will break the work down in JIRA which I will do once I go through
the code and figure out the steps. I will make one top level feature for
the entire change and then make the individual steps tasks. Then everyone
can review the break down comment. If needed I will also put a design doc
together.
Doug
On Wed, Sep 14, 2016 at 8:38 PM, Markus Weimer <ma...@weimo.de> wrote:
> On Wed, Sep 14, 2016 at 7:53 PM, Douglas Service <ds...@gmail.com>
> wrote:
> > Since, no one has objected to replacing Avro with protobuf 3.x, I am
> going
> > to start on that work. First I will spend some time coming to speed on
> the
> > code and understanding the necessary changes, then I will start making
> the
> > changes.
>
>
> +1
>
> Also, it would be great to have some breakdown of work in JIRA for
> this. We could, for example, move through the dependency chain.
>
> Markus
>
Re: [Discuss] Drop Avro in favor of Protobuf 3.x
Posted by Markus Weimer <ma...@weimo.de>.
On Wed, Sep 14, 2016 at 7:53 PM, Douglas Service <ds...@gmail.com> wrote:
> Since, no one has objected to replacing Avro with protobuf 3.x, I am going
> to start on that work. First I will spend some time coming to speed on the
> code and understanding the necessary changes, then I will start making the
> changes.
+1
Also, it would be great to have some breakdown of work in JIRA for
this. We could, for example, move through the dependency chain.
Markus
Re: [Discuss] Drop Avro in favor of Protobuf 3.x
Posted by Douglas Service <ds...@gmail.com>.
Yes I agree, we should do a release just before dropping support for .NET
4.5.
Since, no one has objected to replacing Avro with protobuf 3.x, I am going
to start on that work. First I will spend some time coming to speed on the
code and understanding the necessary changes, then I will start making the
changes.
Doug
On Fri, Sep 9, 2016 at 10:23 AM, Markus Weimer <ma...@weimo.de> wrote:
> On Fri, Sep 9, 2016 at 7:00 AM, Boris Shulman <sh...@gmail.com> wrote:
> > As long as we can compile to .NET 4.5 we should be good.
>
> For the CoreCLR work, we will have to eventually drop support for .NET
> 4.5. I suggest we make one last release of REEF right before doing
> that. Sounds good?
>
> Markus
>
Re: [Discuss] Drop Avro in favor of Protobuf 3.x
Posted by Markus Weimer <ma...@weimo.de>.
On Fri, Sep 9, 2016 at 7:00 AM, Boris Shulman <sh...@gmail.com> wrote:
> As long as we can compile to .NET 4.5 we should be good.
For the CoreCLR work, we will have to eventually drop support for .NET
4.5. I suggest we make one last release of REEF right before doing
that. Sounds good?
Markus
Re: [Discuss] Drop Avro in favor of Protobuf 3.x
Posted by Boris Shulman <sh...@gmail.com>.
As long as we can compile to .NET 4.5 we should be good.
Sent from my iPhone
> On Sep 8, 2016, at 3:21 PM, Markus Weimer <ma...@weimo.de> wrote:
>
> Hi,
>
> protocol buffers recently added .NET support, and Microsoft's Avro
> implementation doesn't seem to go to .NET Core. This is an impediment
> to be on CoreCLR on Linux. Hence, moving all serialization in REEF to
> that new version of protobuf. We have discussed this (or rather:
> hinted at this) in several threads, but I'd like to make this
> discussion and subsequent decision more prominent.
>
> Are there any concerns with this move? We'd remove the use of Avro and
> move to protbuf 3.x for all use in REEF. For Hadoop and YARN, we'll
> have to stick to protobuf 2.5, potentially incurring some shading.
>
> Markus