You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by Douglas Service <ds...@gmail.com> on 2016/09/23 01:23:19 UTC
List of tasks to complete CoreCLR conversion
Here is a list of tasks that appear to be necessary to complete the
conversion to CoreCLR. I would appreciate your feedback so once we have
agreement I can open JIRAs for all of the agreed upon work items. Please
modify the list in place. Items marked as a task would constitute a single
commit and all subitems would need to be completed before a submit is
possible due to dependencies.
- Protobuf 3.0
- Java
- (Task) Move all .proto files used by both java and C# to a
common directory such as lang/common/proto.
- (Task) Enable shading of protobuf 3.0 in maven project files
- (Task) Convert all protobuf usage on java side to 3.0
- Add syntax = "proto2"; to all java .proto files
- Update all calling locations in java code due to changes in
generated code
-
- C#
- (Task) Covert all protobuf usage on C# side to 3.0
- Remove generated files from source tree and git
- Add syntax = "proto2"; to all java .proto files
- Add protoc code generation to C# build
- Update all calling locations in C# code due to changes in
generated code in the following projects (based in inclusion of
protofbuf-net in NuGet package manager):
- REEF.Common
- REEF.Driver
- REEF.Evaluator
- REEF.Evaluator.Tests
- REEF.Network
- REEF.Tang
- REEF.Tang.Tests
- REEF.Tests
- REEF.Wake
- Avro
- (Task) Design and implement an isolation layer that implements the
functionality we use in Avro based on protobuf 3.0.
QUESTION: Is the use of Avro in any of the projects independent of
the other projects such that one could replace the use of Avro
in only that
project with protofuf 3.0 to reduce the size and risk of the commit?
- (Task) Replace Avro in the following projects (based on recursive
grep using namespace names in .avsc files):
- C#
- REEF.Driver
- REEF.Common
- REEF.Tang
- REEF.Client
- java
- reef-bridge-client
- reef-bridge-java
- reef-io
- reef-tang
- reef-runtime
- reef-runtime-mesos
- reef-runtime-multiwebserver
- reef-runtime-hindsight
- reef-runtime-yarn
- reef-examples
- reef-webserver
-
- Nameservice??
- Driver (Task)
- Replace C++ CLI code with protobuf 3.0
- Convert to two processes one C# the other java
Doug
Re: List of tasks to complete CoreCLR conversion
Posted by Douglas Service <ds...@gmail.com>.
You are correct. The release notes for C# clearly say only proto3. Thanks
for catching that issues.
Doug
On Fri, Sep 23, 2016 at 11:08 AM, Markus Weimer <ma...@weimo.de> wrote:
> On 2016-09-23 11:02 AM, Douglas Service wrote:
>
>> Here is what the 3.0 release notes say
>>
>
> Yes, but does this apply to the C# code base only recently merged? Its
> documentation is pretty clear about not supporting `proto2`, even in the
> HEAD of the current `master` branch. [0]
>
> Markus
>
> [0]: https://github.com/google/protobuf/tree/master/csharp
>
Re: List of tasks to complete CoreCLR conversion
Posted by Markus Weimer <ma...@weimo.de>.
On 2016-09-23 11:02 AM, Douglas Service wrote:
> Here is what the 3.0 release notes say
Yes, but does this apply to the C# code base only recently merged? Its
documentation is pretty clear about not supporting `proto2`, even in the
HEAD of the current `master` branch. [0]
Markus
[0]: https://github.com/google/protobuf/tree/master/csharp
Re: List of tasks to complete CoreCLR conversion
Posted by Douglas Service <ds...@gmail.com>.
Here is what the 3.0 release notes say
A new notion "syntax" is introduced to specify whether a .proto file
uses proto2 or proto3:
// foo.proto
syntax = "proto3";
message Bar {...}
If omitted, the protocol buffer compiler generates a warning and "proto2" is
used as the default. This warning will be turned into an error in a future
release.
We eventually want syntax="proto3"; , but if we may be able to use
syntax="proto2"; as an intermediate step
to reduce risk.
Doug
On Fri, Sep 23, 2016 at 10:24 AM, Markus Weimer <ma...@weimo.de> wrote:
> Hi Doug,
>
> the list looks good overall. However, I don't think we can stay on the
> old `proto2` syntax. The documentation for the C# version of protobuf[0]
> states:
>
> The old code only supported proto2; the new code only supports
>> proto3
>>
>
> Hence, we need to account for the conversion of all .proto files to the
> new syntax as well.
>
> On 2016-09-22 6:23 PM, Douglas Service wrote:
>
>> QUESTION: Is the use of Avro in any of the projects independent of
>> the other projects such that one could replace the use of Avro in
>> only that project with protofuf 3.0 to reduce the size and risk of
>> the commit?
>>
>
> Yes, that is generally the case. For example, Tang's use of Avro is
> independent of Wake's and so on.
>
> Markus
>
>
> [0]: https://github.com/google/protobuf/tree/master/csharp
>
>
Re: List of tasks to complete CoreCLR conversion
Posted by Markus Weimer <ma...@weimo.de>.
Hi Doug,
the list looks good overall. However, I don't think we can stay on the
old `proto2` syntax. The documentation for the C# version of protobuf[0]
states:
> The old code only supported proto2; the new code only supports
> proto3
Hence, we need to account for the conversion of all .proto files to the
new syntax as well.
On 2016-09-22 6:23 PM, Douglas Service wrote:
> QUESTION: Is the use of Avro in any of the projects independent of
> the other projects such that one could replace the use of Avro in
> only that project with protofuf 3.0 to reduce the size and risk of
> the commit?
Yes, that is generally the case. For example, Tang's use of Avro is
independent of Wake's and so on.
Markus
[0]: https://github.com/google/protobuf/tree/master/csharp
Re: List of tasks to complete CoreCLR conversion
Posted by Markus Weimer <ma...@weimo.de>.
On 2016-09-23 16:46, Douglas Service wrote:
> Are we going for .NET Core version 1.0 or 1.1 compliance. For
> example, on REEF.Examples we are currently at 100% compatible for
> 1.1 but not for 1.0.
Good question. The roadmap[0] indicates that .NET Core 1.1 should be out
in the "fall". If that holds, we might be able to target it. Are there
beta builds available?
Markus
[0]: https://github.com/dotnet/core/blob/master/roadmap.md
Re: List of tasks to complete CoreCLR conversion
Posted by Douglas Service <ds...@gmail.com>.
Are we going for .NET Core version 1.0 or 1.1 compliance. For example, on
REEF.Examples we are currently at 100% compatible for 1.1 but not for 1.0.
On Fri, Sep 23, 2016 at 3:19 PM, Douglas Service <ds...@gmail.com> wrote:
> One thing I forgot to add is the remaining tasks necessary to remove
> CoreCLR API changes which I am putting together by run the compatibility
> tool now and will send out to the list when I am done.
>
> On Fri, Sep 23, 2016 at 11:47 AM, Douglas Service <ds...@gmail.com>
> wrote:
>
>> Thanks!
>>
>> On Fri, Sep 23, 2016 at 11:38 AM, Mariia Mykhailova <
>> mamykhai@microsoft.com> wrote:
>>
>>> I'll do a first iteration of closing Avro-related JIRAs (for example,
>>> REEF-1536 "Enable Avro pretty print" - we probably don't want to invest
>>> into making Avro pretty if we'll be getting rid of it entirely soon). Then
>>> anybody can re-open ones which might still have some value and close any
>>> ones I've missed.
>>>
>>> -Mariia
>>>
>>> -----Original Message-----
>>> From: Markus Weimer [mailto:markus@weimo.de]
>>> Sent: Friday, September 23, 2016 11:21 AM
>>> To: dev@reef.apache.org
>>> Subject: Re: List of tasks to complete CoreCLR conversion
>>>
>>> On 2016-09-23 11:11 AM, Mariia Mykhailova wrote:
>>> > We should also go through JIRAs we filed for migration from protobuf
>>> > to Avro and close them to reduce confusion in any potential reader
>>> > :-)
>>>
>>> +1
>>>
>>> Markus
>>>
>>
>>
>
Re: List of tasks to complete CoreCLR conversion
Posted by Douglas Service <ds...@gmail.com>.
One thing I forgot to add is the remaining tasks necessary to remove
CoreCLR API changes which I am putting together by run the compatibility
tool now and will send out to the list when I am done.
On Fri, Sep 23, 2016 at 11:47 AM, Douglas Service <ds...@gmail.com> wrote:
> Thanks!
>
> On Fri, Sep 23, 2016 at 11:38 AM, Mariia Mykhailova <
> mamykhai@microsoft.com> wrote:
>
>> I'll do a first iteration of closing Avro-related JIRAs (for example,
>> REEF-1536 "Enable Avro pretty print" - we probably don't want to invest
>> into making Avro pretty if we'll be getting rid of it entirely soon). Then
>> anybody can re-open ones which might still have some value and close any
>> ones I've missed.
>>
>> -Mariia
>>
>> -----Original Message-----
>> From: Markus Weimer [mailto:markus@weimo.de]
>> Sent: Friday, September 23, 2016 11:21 AM
>> To: dev@reef.apache.org
>> Subject: Re: List of tasks to complete CoreCLR conversion
>>
>> On 2016-09-23 11:11 AM, Mariia Mykhailova wrote:
>> > We should also go through JIRAs we filed for migration from protobuf
>> > to Avro and close them to reduce confusion in any potential reader
>> > :-)
>>
>> +1
>>
>> Markus
>>
>
>
Re: List of tasks to complete CoreCLR conversion
Posted by Douglas Service <ds...@gmail.com>.
Thanks!
On Fri, Sep 23, 2016 at 11:38 AM, Mariia Mykhailova <ma...@microsoft.com>
wrote:
> I'll do a first iteration of closing Avro-related JIRAs (for example,
> REEF-1536 "Enable Avro pretty print" - we probably don't want to invest
> into making Avro pretty if we'll be getting rid of it entirely soon). Then
> anybody can re-open ones which might still have some value and close any
> ones I've missed.
>
> -Mariia
>
> -----Original Message-----
> From: Markus Weimer [mailto:markus@weimo.de]
> Sent: Friday, September 23, 2016 11:21 AM
> To: dev@reef.apache.org
> Subject: Re: List of tasks to complete CoreCLR conversion
>
> On 2016-09-23 11:11 AM, Mariia Mykhailova wrote:
> > We should also go through JIRAs we filed for migration from protobuf
> > to Avro and close them to reduce confusion in any potential reader
> > :-)
>
> +1
>
> Markus
>
RE: List of tasks to complete CoreCLR conversion
Posted by Mariia Mykhailova <ma...@microsoft.com>.
I'll do a first iteration of closing Avro-related JIRAs (for example, REEF-1536 "Enable Avro pretty print" - we probably don't want to invest into making Avro pretty if we'll be getting rid of it entirely soon). Then anybody can re-open ones which might still have some value and close any ones I've missed.
-Mariia
-----Original Message-----
From: Markus Weimer [mailto:markus@weimo.de]
Sent: Friday, September 23, 2016 11:21 AM
To: dev@reef.apache.org
Subject: Re: List of tasks to complete CoreCLR conversion
On 2016-09-23 11:11 AM, Mariia Mykhailova wrote:
> We should also go through JIRAs we filed for migration from protobuf
> to Avro and close them to reduce confusion in any potential reader
> :-)
+1
Markus
Re: List of tasks to complete CoreCLR conversion
Posted by Markus Weimer <ma...@weimo.de>.
On 2016-09-23 11:11 AM, Mariia Mykhailova wrote:
> We should also go through JIRAs we filed for migration from protobuf
> to Avro and close them to reduce confusion in any potential reader
> :-)
+1
Markus
RE: List of tasks to complete CoreCLR conversion
Posted by Mariia Mykhailova <ma...@microsoft.com>.
We should also go through JIRAs we filed for migration from protobuf to Avro and close them to reduce confusion in any potential reader :-)
-Mariia
-----Original Message-----
From: Douglas Service [mailto:dsopsrc@gmail.com]
Sent: Thursday, September 22, 2016 6:23 PM
To: dev@reef.apache.org
Subject: List of tasks to complete CoreCLR conversion
Here is a list of tasks that appear to be necessary to complete the conversion to CoreCLR. I would appreciate your feedback so once we have agreement I can open JIRAs for all of the agreed upon work items. Please modify the list in place. Items marked as a task would constitute a single commit and all subitems would need to be completed before a submit is possible due to dependencies.
- Protobuf 3.0
- Java
- (Task) Move all .proto files used by both java and C# to a
common directory such as lang/common/proto.
- (Task) Enable shading of protobuf 3.0 in maven project files
- (Task) Convert all protobuf usage on java side to 3.0
- Add syntax = "proto2"; to all java .proto files
- Update all calling locations in java code due to changes in
generated code
-
- C#
- (Task) Covert all protobuf usage on C# side to 3.0
- Remove generated files from source tree and git
- Add syntax = "proto2"; to all java .proto files
- Add protoc code generation to C# build
- Update all calling locations in C# code due to changes in
generated code in the following projects (based in inclusion of
protofbuf-net in NuGet package manager):
- REEF.Common
- REEF.Driver
- REEF.Evaluator
- REEF.Evaluator.Tests
- REEF.Network
- REEF.Tang
- REEF.Tang.Tests
- REEF.Tests
- REEF.Wake
- Avro
- (Task) Design and implement an isolation layer that implements the
functionality we use in Avro based on protobuf 3.0.
QUESTION: Is the use of Avro in any of the projects independent of
the other projects such that one could replace the use of Avro in only that
project with protofuf 3.0 to reduce the size and risk of the commit?
- (Task) Replace Avro in the following projects (based on recursive
grep using namespace names in .avsc files):
- C#
- REEF.Driver
- REEF.Common
- REEF.Tang
- REEF.Client
- java
- reef-bridge-client
- reef-bridge-java
- reef-io
- reef-tang
- reef-runtime
- reef-runtime-mesos
- reef-runtime-multiwebserver
- reef-runtime-hindsight
- reef-runtime-yarn
- reef-examples
- reef-webserver
-
- Nameservice??
- Driver (Task)
- Replace C++ CLI code with protobuf 3.0
- Convert to two processes one C# the other java
Doug