Protobuf version upgrade?

[Byung-Gon Chun <bg...@gmail.com>]

Hi,

We still depends on protobuf 2.5. Can we upgrade it? Protobuf 3.x is already available.

-Gon

Sent from my iPhone

Re: Protobuf version upgrade?

[Gyewon Lee <st...@gmail.com>]

Thank you for response, Douglas. That plan seems good to me.

Best regards,
Gyewon Lee

2016-10-18 8:21 GMT+09:00 Douglas Service <ds...@gmail.com>:

> As a first estimate I would expect that type of functionally to begin
> coming online by the end of the year. It depends on how quickly the work
> goes the extent of which I am still investigating.
>
> Doug
>
> On Thu, Oct 13, 2016 at 8:11 PM, Gyewon Lee <st...@gmail.com> wrote:
>
> > Hi, Douglas.
> >
> > Thank you a lot for notifying us.
> >
> > Actually, there was long lasting need for using some of the new features
> of
> > Apache Avro (like Union) in C# Reef, especially in Tang. How long will it
> > take for C# Reef to support the latest schema of Apache Avro according to
> > this plan?
> >
> > Best regards,
> > Gyewon Lee
> >
> > 2016-10-14 10:31 GMT+09:00 Douglas Service <ds...@gmail.com>:
> >
> > > There is an announcement for new version of .NET called .NET standard
> 2.0
> > > that appears to follow .NET Core fairly soon https://blogs.msdn.
> > > microsoft.com/dotnet/2016/09/26/introducing-net-standard/ which would
> > make
> > > it much easier for us to continue using Avro with C# and remove the
> need
> > to
> > > convert all of our current usage of Avro to protobuf. We would most
> > likely
> > > need to pull the code base for the C# version of Avro we currently use
> > into
> > > REEF, fix a small number of compatibility issues (much smaller than for
> > > .NET Core 1.0 or 1.1), and then try to eventually get the features we
> > need
> > > which are different from the Apache Avro C# API merged into Apache
> Avro.
> > >
> > > Thoughts?
> > >
> > > Doug
> > >
> >
>
>
>
> --
>
> On Mon, Oct 3, 2016 at 5:50 PM, Markus Weimer <ma...@weimo.de> wrote:
> On Mon, Oct 3, 2016 at 4:55 PM, Douglas Service <ds...@gmail.com> wrote:
> > Questions:
> > 1) If the extra messages are not used by both sides they should probably
> > not be in this file. Will these messages be shared in the futur
>

Re: Protobuf version upgrade?

[Douglas Service <ds...@gmail.com>]

As a first estimate I would expect that type of functionally to begin
coming online by the end of the year. It depends on how quickly the work
goes the extent of which I am still investigating.

Doug

On Thu, Oct 13, 2016 at 8:11 PM, Gyewon Lee <st...@gmail.com> wrote:

> Hi, Douglas.
>
> Thank you a lot for notifying us.
>
> Actually, there was long lasting need for using some of the new features of
> Apache Avro (like Union) in C# Reef, especially in Tang. How long will it
> take for C# Reef to support the latest schema of Apache Avro according to
> this plan?
>
> Best regards,
> Gyewon Lee
>
> 2016-10-14 10:31 GMT+09:00 Douglas Service <ds...@gmail.com>:
>
> > There is an announcement for new version of .NET called .NET standard 2.0
> > that appears to follow .NET Core fairly soon https://blogs.msdn.
> > microsoft.com/dotnet/2016/09/26/introducing-net-standard/ which would
> make
> > it much easier for us to continue using Avro with C# and remove the need
> to
> > convert all of our current usage of Avro to protobuf. We would most
> likely
> > need to pull the code base for the C# version of Avro we currently use
> into
> > REEF, fix a small number of compatibility issues (much smaller than for
> > .NET Core 1.0 or 1.1), and then try to eventually get the features we
> need
> > which are different from the Apache Avro C# API merged into Apache Avro.
> >
> > Thoughts?
> >
> > Doug
> >
>



-- 

On Mon, Oct 3, 2016 at 5:50 PM, Markus Weimer <ma...@weimo.de> wrote:
On Mon, Oct 3, 2016 at 4:55 PM, Douglas Service <ds...@gmail.com> wrote:
> Questions:
> 1) If the extra messages are not used by both sides they should probably
> not be in this file. Will these messages be shared in the futur

Re: Protobuf version upgrade?

[Gyewon Lee <st...@gmail.com>]

Hi, Douglas.

Thank you a lot for notifying us.

Actually, there was long lasting need for using some of the new features of
Apache Avro (like Union) in C# Reef, especially in Tang. How long will it
take for C# Reef to support the latest schema of Apache Avro according to
this plan?

Best regards,
Gyewon Lee

2016-10-14 10:31 GMT+09:00 Douglas Service <ds...@gmail.com>:

> There is an announcement for new version of .NET called .NET standard 2.0
> that appears to follow .NET Core fairly soon https://blogs.msdn.
> microsoft.com/dotnet/2016/09/26/introducing-net-standard/ which would make
> it much easier for us to continue using Avro with C# and remove the need to
> convert all of our current usage of Avro to protobuf. We would most likely
> need to pull the code base for the C# version of Avro we currently use into
> REEF, fix a small number of compatibility issues (much smaller than for
> .NET Core 1.0 or 1.1), and then try to eventually get the features we need
> which are different from the Apache Avro C# API merged into Apache Avro.
>
> Thoughts?
>
> Doug
>

Re: Protobuf version upgrade?

[Boris Shulman <sh...@gmail.com>]

The question is what .NET core version will support .NET standard 2.0 and when it will be available.

Sent from my iPhone

> On Oct 13, 2016, at 6:31 PM, Douglas Service <ds...@gmail.com> wrote:
> 
> There is an announcement for new version of .NET called .NET standard 2.0
> that appears to follow .NET Core fairly soon https://blogs.msdn.
> microsoft.com/dotnet/2016/09/26/introducing-net-standard/ which would make
> it much easier for us to continue using Avro with C# and remove the need to
> convert all of our current usage of Avro to protobuf. We would most likely
> need to pull the code base for the C# version of Avro we currently use into
> REEF, fix a small number of compatibility issues (much smaller than for
> .NET Core 1.0 or 1.1), and then try to eventually get the features we need
> which are different from the Apache Avro C# API merged into Apache Avro.
> 
> Thoughts?
> 
> Doug

Re: Protobuf version upgrade?

[Douglas Service <ds...@gmail.com>]

There is an announcement for new version of .NET called .NET standard 2.0
that appears to follow .NET Core fairly soon https://blogs.msdn.
microsoft.com/dotnet/2016/09/26/introducing-net-standard/ which would make
it much easier for us to continue using Avro with C# and remove the need to
convert all of our current usage of Avro to protobuf. We would most likely
need to pull the code base for the C# version of Avro we currently use into
REEF, fix a small number of compatibility issues (much smaller than for
.NET Core 1.0 or 1.1), and then try to eventually get the features we need
which are different from the Apache Avro C# API merged into Apache Avro.

Thoughts?

Doug

Re: Protobuf version upgrade?

[Byung-Gon Chun <bg...@gmail.com>]

Oh. I see.
I am fine with going with protobuf.

-Gon

On Thu, Sep 8, 2016 at 2:48 PM, Markus Weimer <ma...@weimo.de> wrote:

> Yeah, this is a bit of a mess. We initially pushed for Avro because
> (a) the main version is from Apache and (b) there is a .NET version
> from Microsoft available under the ASL. Protobuf did not exist for
> .NET at the time. It now does, and the Microsoft implementation seems
> abandoned and won't make it to CoreCLR.
>
> Hence, we are flip-flopping on the question of what the best
> serializer for REEF's use case is. Right now, protobuf seems like the
> only one available for Java and .NET (including CoreCLR).
>
> Markus
>
> On Thu, Sep 8, 2016 at 2:45 PM, Byung-Gon Chun <bg...@gmail.com> wrote:
> > Douglas,
> >
> > Do we want to completely remove the Avro dependency?
> > At some point, we decided to take the Avro dependency by removing
> protobuf.
> >
> > -Gon
> >
> > On Thu, Sep 8, 2016 at 2:30 PM, Douglas Service <ds...@gmail.com>
> wrote:
> >
> >> This appears to be best way to remove the Avro dependency as we port to
> >> .NET Core.
> >>
> >> On Wed, Sep 7, 2016 at 8:29 AM, Markus Weimer <ma...@weimo.de> wrote:
> >>
> >> > On 2016-09-02 3:34 PM, Douglas Service wrote:
> >> >
> >> >> http://stackoverflow.com/questions/34487996/how-can-i-use-
> >> >> proto3-with-hadoop-spark
> >> >>
> >> >
> >> > The top answer there is to use Shading in the resulting JAR. That
> works,
> >> > but comes with all the downsides of shading. I'm OK with this, as
> long as
> >> > we document this really well on our website such that our users know
> if /
> >> > when to shade protobuf in their own builds.
> >> >
> >> > What do others think?
> >> >
> >> > Markus
> >> >
> >>
> >
> >
> >
> > --
> > Byung-Gon Chun
>



-- 
Byung-Gon Chun

Re: Protobuf version upgrade?

[Markus Weimer <ma...@weimo.de>]

Yeah, this is a bit of a mess. We initially pushed for Avro because
(a) the main version is from Apache and (b) there is a .NET version
from Microsoft available under the ASL. Protobuf did not exist for
.NET at the time. It now does, and the Microsoft implementation seems
abandoned and won't make it to CoreCLR.

Hence, we are flip-flopping on the question of what the best
serializer for REEF's use case is. Right now, protobuf seems like the
only one available for Java and .NET (including CoreCLR).

Markus

On Thu, Sep 8, 2016 at 2:45 PM, Byung-Gon Chun <bg...@gmail.com> wrote:
> Douglas,
>
> Do we want to completely remove the Avro dependency?
> At some point, we decided to take the Avro dependency by removing protobuf.
>
> -Gon
>
> On Thu, Sep 8, 2016 at 2:30 PM, Douglas Service <ds...@gmail.com> wrote:
>
>> This appears to be best way to remove the Avro dependency as we port to
>> .NET Core.
>>
>> On Wed, Sep 7, 2016 at 8:29 AM, Markus Weimer <ma...@weimo.de> wrote:
>>
>> > On 2016-09-02 3:34 PM, Douglas Service wrote:
>> >
>> >> http://stackoverflow.com/questions/34487996/how-can-i-use-
>> >> proto3-with-hadoop-spark
>> >>
>> >
>> > The top answer there is to use Shading in the resulting JAR. That works,
>> > but comes with all the downsides of shading. I'm OK with this, as long as
>> > we document this really well on our website such that our users know if /
>> > when to shade protobuf in their own builds.
>> >
>> > What do others think?
>> >
>> > Markus
>> >
>>
>
>
>
> --
> Byung-Gon Chun

Re: Protobuf version upgrade?

[Byung-Gon Chun <bg...@gmail.com>]

Douglas,

Do we want to completely remove the Avro dependency?
At some point, we decided to take the Avro dependency by removing protobuf.

-Gon

On Thu, Sep 8, 2016 at 2:30 PM, Douglas Service <ds...@gmail.com> wrote:

> This appears to be best way to remove the Avro dependency as we port to
> .NET Core.
>
> On Wed, Sep 7, 2016 at 8:29 AM, Markus Weimer <ma...@weimo.de> wrote:
>
> > On 2016-09-02 3:34 PM, Douglas Service wrote:
> >
> >> http://stackoverflow.com/questions/34487996/how-can-i-use-
> >> proto3-with-hadoop-spark
> >>
> >
> > The top answer there is to use Shading in the resulting JAR. That works,
> > but comes with all the downsides of shading. I'm OK with this, as long as
> > we document this really well on our website such that our users know if /
> > when to shade protobuf in their own builds.
> >
> > What do others think?
> >
> > Markus
> >
>



-- 
Byung-Gon Chun

Re: Protobuf version upgrade?

[Douglas Service <ds...@gmail.com>]

This appears to be best way to remove the Avro dependency as we port to
.NET Core.

On Wed, Sep 7, 2016 at 8:29 AM, Markus Weimer <ma...@weimo.de> wrote:

> On 2016-09-02 3:34 PM, Douglas Service wrote:
>
>> http://stackoverflow.com/questions/34487996/how-can-i-use-
>> proto3-with-hadoop-spark
>>
>
> The top answer there is to use Shading in the resulting JAR. That works,
> but comes with all the downsides of shading. I'm OK with this, as long as
> we document this really well on our website such that our users know if /
> when to shade protobuf in their own builds.
>
> What do others think?
>
> Markus
>

Re: Protobuf version upgrade?

[Markus Weimer <ma...@weimo.de>]

On 2016-09-02 3:34 PM, Douglas Service wrote:
> http://stackoverflow.com/questions/34487996/how-can-i-use-proto3-with-hadoop-spark

The top answer there is to use Shading in the resulting JAR. That works, 
but comes with all the downsides of shading. I'm OK with this, as long 
as we document this really well on our website such that our users know 
if / when to shade protobuf in their own builds.

What do others think?

Markus

Re: Protobuf version upgrade?

[Douglas Service <ds...@gmail.com>]

http://stackoverflow.com/questions/34487996/how-can-i-use-proto3-with-hadoop-spark

On Fri, Sep 2, 2016 at 8:51 AM, Markus Weimer <ma...@weimo.de> wrote:

> On 2016-09-01 17:39, Douglas Service wrote:
>
>> Mario pointed me to the following link [...]
>>
>
> The link did not make it through. Can you post it in plain text?
>
> Thanks,
>
> Markus
>

Re: Protobuf version upgrade?

[Markus Weimer <ma...@weimo.de>]

On 2016-09-01 17:39, Douglas Service wrote:
> Mario pointed me to the following link [...]

The link did not make it through. Can you post it in plain text?

Thanks,

Markus

Re: Protobuf version upgrade?

[Douglas Service <ds...@gmail.com>]

I agree. We need to upgrade to Protobuf 3.x. Mario pointed me to the
following link that shows how to use both Protobuf 2.5 and 3.x together
which would allow us to use 2.5 with Yarn and change our internal usage to
3.0 and replace Avro. Thoughts?

Doug

On Wed, Aug 17, 2016 at 4:08 PM, Markus Weimer <ma...@weimo.de> wrote:

> On Wed, Aug 17, 2016 at 11:38 AM, Byung-Gon Chun <bg...@gmail.com> wrote:
> >
> > We still depends on protobuf 2.5. Can we upgrade it? Protobuf 3.x is
> already available.
>
>
> I'd love to, especially as that version has native support for .NET.
> However, I'm not sure how this plays with YARN, which is still at
> version 2.5. If we can show that upgrading won't break our ability to
> run on YARN, we should.
>
> Markus
>

Re: Protobuf version upgrade?

[Markus Weimer <ma...@weimo.de>]

On Wed, Aug 17, 2016 at 11:38 AM, Byung-Gon Chun <bg...@gmail.com> wrote:
>
> We still depends on protobuf 2.5. Can we upgrade it? Protobuf 3.x is already available.


I'd love to, especially as that version has native support for .NET.
However, I'm not sure how this plays with YARN, which is still at
version 2.5. If we can show that upgrading won't break our ability to
run on YARN, we should.

Markus