You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Olivier Sallou <ol...@irisa.fr> on 2017/03/23 08:54:23 UTC
protbuf to json not compatible
Hi,
when transforming a protobug message to json with MessageToJson, the
json is not compatible with the json format expected by Mesos master.
For example, for volumes it generates
volumes: [
{'hostPath': '....',
'containerPath': '...',
...
}
]
but HTTP API expects "source" and "container_path"
is it an expected behavior ? This prevents from "creating" a task in
protobuf format and sending it to HTTP API with a protobug to json
conversion.
Thanks
Olivier
--
Olivier Sallou
IRISA / University of Rennes 1
Campus de Beaulieu, 35000 RENNES - FRANCE
Tel: 02.99.84.71.95
gpg key id: 4096R/326D8438 (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
Re : Re: protbuf to json not compatible
Posted by Olivier Sallou <ol...@irisa.fr>.
----- Benjamin Mahler <bm...@apache.org> a écrit :
> James, I'm curious, do you know specifically what the incompatibility is?
>
> Olivier, if you're dealing with protobuf already and trying to send it to
> mesos, there's no need to use JSON. Unless you have a requirement to do so?
I can manage json, this is fine.
Sending protobuf mean sending whole accept message as protobuf, not task definition only. But for this I need mesos.native python package, and i want to avoid this. So i will switch to full json.
Olivier
> There are some outstanding issues with our JSON<->Protobuf conversion,
> specifically we currently are inconsistent from proto3 when it comes to the
> int(32|64), fixed(32|64), uint(32|64) handling, for one (we don't allow
> strings on the input side (tomek is addressing that), and we don't use
> strings on the output side).
>
> On Fri, Mar 24, 2017 at 12:44 AM, Olivier Sallou <ol...@irisa.fr>
> wrote:
>
> >
> >
> > On 03/24/2017 04:02 AM, James Peach wrote:
> > >> On Mar 23, 2017, at 7:58 PM, James Peach <jo...@gmail.com> wrote:
> > >>
> > >>> On Mar 23, 2017, at 1:54 AM, Olivier Sallou <ol...@irisa.fr>
> > wrote:
> > >>>
> > >>> Hi,
> > >>>
> > >>> when transforming a protobug message to json with MessageToJson, the
> > >>> json is not compatible with the json format expected by Mesos master.
> > >> This is because you generated the protobuf bindings with proto3
> > compiler. AFAICT they made an incompatible change to the JSON wire format.
> > This bites you when using the jsonpb Go package, for example. I ended up
> > post-processing the generated Go code to correct the field names.
> > > Sorry I forgot to mention that the other workaround is to generate the
> > protobuf bindings with the proto2 compiler.
> > Thanks
> > My first workaround is to generate json directly, not a big deal in my
> > case, but I wanted to understand.
> >
> > Olivier
> > >
> > > J
> >
> > --
> > Olivier Sallou
> > IRISA / University of Rennes 1
> > Campus de Beaulieu, 35000 RENNES - FRANCE
> > Tel: 02.99.84.71.95
> >
> > gpg key id: 4096R/326D8438 (keyring.debian.org)
> > Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
> >
> >
Re: Re : Re: protbuf to json not compatible
Posted by Benjamin Mahler <bm...@apache.org>.
As a note, if you just need the protobufs you should be able to run protoc
against the published protos for the version you're building against
without needing to pull in the 'mesos.native' package. The 'mesos.native'
package provides scheduler and executor implementations (via talking to the
C++ mesos libs), and it sounds like you just need the .proto files?
I'm curious how other projects deal with "publishing" protobuf, whether
they publish packages compiled by "protoc" per language or just publish the
.protos. Seems like the latter is the more reasonable one for projects to
do.
On Sat, Mar 25, 2017 at 12:40 AM, Olivier Sallou <ol...@irisa.fr>
wrote:
>
> ----- Benjamin Mahler <bm...@apache.org> a écrit :
> > James, I'm curious, do you know specifically what the incompatibility is?
> >
> > Olivier, if you're dealing with protobuf already and trying to send it to
> > mesos, there's no need to use JSON. Unless you have a requirement to do
> so?
>
> I can manage json, this is fine.
> Sending protobuf mean sending whole accept message as protobuf, not task
> definition only. But for this I need mesos.native python package, and i
> want to avoid this. So i will switch to full json.
>
> Olivier
>
>
>
> > There are some outstanding issues with our JSON<->Protobuf conversion,
> > specifically we currently are inconsistent from proto3 when it comes to
> the
> > int(32|64), fixed(32|64), uint(32|64) handling, for one (we don't allow
> > strings on the input side (tomek is addressing that), and we don't use
> > strings on the output side).
> >
> > On Fri, Mar 24, 2017 at 12:44 AM, Olivier Sallou <
> olivier.sallou@irisa.fr>
> > wrote:
> >
> > >
> > >
> > > On 03/24/2017 04:02 AM, James Peach wrote:
> > > >> On Mar 23, 2017, at 7:58 PM, James Peach <jo...@gmail.com> wrote:
> > > >>
> > > >>> On Mar 23, 2017, at 1:54 AM, Olivier Sallou <
> olivier.sallou@irisa.fr>
> > > wrote:
> > > >>>
> > > >>> Hi,
> > > >>>
> > > >>> when transforming a protobug message to json with MessageToJson,
> the
> > > >>> json is not compatible with the json format expected by Mesos
> master.
> > > >> This is because you generated the protobuf bindings with proto3
> > > compiler. AFAICT they made an incompatible change to the JSON wire
> format.
> > > This bites you when using the jsonpb Go package, for example. I ended
> up
> > > post-processing the generated Go code to correct the field names.
> > > > Sorry I forgot to mention that the other workaround is to generate
> the
> > > protobuf bindings with the proto2 compiler.
> > > Thanks
> > > My first workaround is to generate json directly, not a big deal in my
> > > case, but I wanted to understand.
> > >
> > > Olivier
> > > >
> > > > J
> > >
> > > --
> > > Olivier Sallou
> > > IRISA / University of Rennes 1
> > > Campus de Beaulieu, 35000 RENNES - FRANCE
> > > Tel: 02.99.84.71.95
> > >
> > > gpg key id: 4096R/326D8438 (keyring.debian.org)
> > > Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
> > >
> > >
>
>
Re: protbuf to json not compatible
Posted by James Peach <jo...@gmail.com>.
> On Mar 24, 2017, at 12:49 PM, Benjamin Mahler <bm...@apache.org> wrote:
>
> James, I'm curious, do you know specifically what the incompatibility is?
https://developers.google.com/protocol-buffers/docs/proto3#json
"Message field names are mapped to lowerCamelCase and become JSON object keys. "
So field names like "failover_timeout" are renamed to "failoverTimeout".
>
> Olivier, if you're dealing with protobuf already and trying to send it to
> mesos, there's no need to use JSON. Unless you have a requirement to do so?
> There are some outstanding issues with our JSON<->Protobuf conversion,
> specifically we currently are inconsistent from proto3 when it comes to the
> int(32|64), fixed(32|64), uint(32|64) handling, for one (we don't allow
> strings on the input side (tomek is addressing that), and we don't use
> strings on the output side).
>
> On Fri, Mar 24, 2017 at 12:44 AM, Olivier Sallou <ol...@irisa.fr>
> wrote:
>
>>
>>
>> On 03/24/2017 04:02 AM, James Peach wrote:
>>>> On Mar 23, 2017, at 7:58 PM, James Peach <jo...@gmail.com> wrote:
>>>>
>>>>> On Mar 23, 2017, at 1:54 AM, Olivier Sallou <ol...@irisa.fr>
>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> when transforming a protobug message to json with MessageToJson, the
>>>>> json is not compatible with the json format expected by Mesos master.
>>>> This is because you generated the protobuf bindings with proto3
>> compiler. AFAICT they made an incompatible change to the JSON wire format.
>> This bites you when using the jsonpb Go package, for example. I ended up
>> post-processing the generated Go code to correct the field names.
>>> Sorry I forgot to mention that the other workaround is to generate the
>> protobuf bindings with the proto2 compiler.
>> Thanks
>> My first workaround is to generate json directly, not a big deal in my
>> case, but I wanted to understand.
>>
>> Olivier
>>>
>>> J
>>
>> --
>> Olivier Sallou
>> IRISA / University of Rennes 1
>> Campus de Beaulieu, 35000 RENNES - FRANCE
>> Tel: 02.99.84.71.95
>>
>> gpg key id: 4096R/326D8438 (keyring.debian.org)
>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>>
>>
Re: protbuf to json not compatible
Posted by Benjamin Mahler <bm...@apache.org>.
James, I'm curious, do you know specifically what the incompatibility is?
Olivier, if you're dealing with protobuf already and trying to send it to
mesos, there's no need to use JSON. Unless you have a requirement to do so?
There are some outstanding issues with our JSON<->Protobuf conversion,
specifically we currently are inconsistent from proto3 when it comes to the
int(32|64), fixed(32|64), uint(32|64) handling, for one (we don't allow
strings on the input side (tomek is addressing that), and we don't use
strings on the output side).
On Fri, Mar 24, 2017 at 12:44 AM, Olivier Sallou <ol...@irisa.fr>
wrote:
>
>
> On 03/24/2017 04:02 AM, James Peach wrote:
> >> On Mar 23, 2017, at 7:58 PM, James Peach <jo...@gmail.com> wrote:
> >>
> >>> On Mar 23, 2017, at 1:54 AM, Olivier Sallou <ol...@irisa.fr>
> wrote:
> >>>
> >>> Hi,
> >>>
> >>> when transforming a protobug message to json with MessageToJson, the
> >>> json is not compatible with the json format expected by Mesos master.
> >> This is because you generated the protobuf bindings with proto3
> compiler. AFAICT they made an incompatible change to the JSON wire format.
> This bites you when using the jsonpb Go package, for example. I ended up
> post-processing the generated Go code to correct the field names.
> > Sorry I forgot to mention that the other workaround is to generate the
> protobuf bindings with the proto2 compiler.
> Thanks
> My first workaround is to generate json directly, not a big deal in my
> case, but I wanted to understand.
>
> Olivier
> >
> > J
>
> --
> Olivier Sallou
> IRISA / University of Rennes 1
> Campus de Beaulieu, 35000 RENNES - FRANCE
> Tel: 02.99.84.71.95
>
> gpg key id: 4096R/326D8438 (keyring.debian.org)
> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>
>
Re: protbuf to json not compatible
Posted by Olivier Sallou <ol...@irisa.fr>.
On 03/24/2017 04:02 AM, James Peach wrote:
>> On Mar 23, 2017, at 7:58 PM, James Peach <jo...@gmail.com> wrote:
>>
>>> On Mar 23, 2017, at 1:54 AM, Olivier Sallou <ol...@irisa.fr> wrote:
>>>
>>> Hi,
>>>
>>> when transforming a protobug message to json with MessageToJson, the
>>> json is not compatible with the json format expected by Mesos master.
>> This is because you generated the protobuf bindings with proto3 compiler. AFAICT they made an incompatible change to the JSON wire format. This bites you when using the jsonpb Go package, for example. I ended up post-processing the generated Go code to correct the field names.
> Sorry I forgot to mention that the other workaround is to generate the protobuf bindings with the proto2 compiler.
Thanks
My first workaround is to generate json directly, not a big deal in my
case, but I wanted to understand.
Olivier
>
> J
--
Olivier Sallou
IRISA / University of Rennes 1
Campus de Beaulieu, 35000 RENNES - FRANCE
Tel: 02.99.84.71.95
gpg key id: 4096R/326D8438 (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
Re: protbuf to json not compatible
Posted by James Peach <jo...@gmail.com>.
> On Mar 23, 2017, at 7:58 PM, James Peach <jo...@gmail.com> wrote:
>
>>
>> On Mar 23, 2017, at 1:54 AM, Olivier Sallou <ol...@irisa.fr> wrote:
>>
>> Hi,
>>
>> when transforming a protobug message to json with MessageToJson, the
>> json is not compatible with the json format expected by Mesos master.
>
> This is because you generated the protobuf bindings with proto3 compiler. AFAICT they made an incompatible change to the JSON wire format. This bites you when using the jsonpb Go package, for example. I ended up post-processing the generated Go code to correct the field names.
Sorry I forgot to mention that the other workaround is to generate the protobuf bindings with the proto2 compiler.
J
Re: protbuf to json not compatible
Posted by James Peach <jo...@gmail.com>.
> On Mar 23, 2017, at 1:54 AM, Olivier Sallou <ol...@irisa.fr> wrote:
>
> Hi,
>
> when transforming a protobug message to json with MessageToJson, the
> json is not compatible with the json format expected by Mesos master.
This is because you generated the protobuf bindings with proto3 compiler. AFAICT they made an incompatible change to the JSON wire format. This bites you when using the jsonpb Go package, for example. I ended up post-processing the generated Go code to correct the field names.
>
> For example, for volumes it generates
>
>
> volumes: [
>
> {'hostPath': '....',
>
> 'containerPath': '...',
>
> ...
>
> }
>
> ]
>
>
> but HTTP API expects "source" and "container_path"
>
> is it an expected behavior ? This prevents from "creating" a task in
> protobuf format and sending it to HTTP API with a protobug to json
> conversion.
It’s expected from a proto3 compiler. IMHO this is a breaking change and they should have made a fallback option, but there’s not one. There’s no good choices here AFAICT.
J
Re: protbuf to json not compatible
Posted by Benjamin Mahler <bm...@apache.org>.
+gilbert
On Thu, Mar 23, 2017 at 1:54 AM, Olivier Sallou <ol...@irisa.fr>
wrote:
> Hi,
>
> when transforming a protobug message to json with MessageToJson, the
> json is not compatible with the json format expected by Mesos master.
>
> For example, for volumes it generates
>
>
> volumes: [
>
> {'hostPath': '....',
>
> 'containerPath': '...',
>
> ...
>
> }
>
> ]
>
>
> but HTTP API expects "source" and "container_path"
>
> is it an expected behavior ? This prevents from "creating" a task in
> protobuf format and sending it to HTTP API with a protobug to json
> conversion.
>
> Thanks
>
> Olivier
>
> --
> Olivier Sallou
> IRISA / University of Rennes 1
> Campus de Beaulieu, 35000 RENNES - FRANCE
> Tel: 02.99.84.71.95
>
> gpg key id: 4096R/326D8438 (keyring.debian.org)
> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>
>
Re: protbuf to json not compatible
Posted by Benjamin Mahler <bm...@apache.org>.
+anand, vinod
On Thu, Mar 23, 2017 at 2:09 AM, Tomek Janiszewski <ja...@gmail.com>
wrote:
> I have a similar problem with protobuf and json. In my case numbers were
> incorrectly unmarshaled. Here is an issue
> https://issues.apache.org/jira/browse/MESOS-970 and review
> https://reviews.apache.org/r/50851/
>
> czw., 23.03.2017, 09:54 użytkownik Olivier Sallou <olivier.sallou@irisa.fr
> >
> napisał:
>
> > Hi,
> >
> > when transforming a protobug message to json with MessageToJson, the
> > json is not compatible with the json format expected by Mesos master.
> >
> > For example, for volumes it generates
> >
> >
> > volumes: [
> >
> > {'hostPath': '....',
> >
> > 'containerPath': '...',
> >
> > ...
> >
> > }
> >
> > ]
> >
> >
> > but HTTP API expects "source" and "container_path"
> >
> > is it an expected behavior ? This prevents from "creating" a task in
> > protobuf format and sending it to HTTP API with a protobug to json
> > conversion.
> >
> > Thanks
> >
> > Olivier
> >
> > --
> > Olivier Sallou
> > IRISA / University of Rennes 1
> > Campus de Beaulieu, 35000 RENNES - FRANCE
> > Tel: 02.99.84.71.95
> >
> > gpg key id: 4096R/326D8438 (keyring.debian.org)
> > Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
> >
> >
>
Re: protbuf to json not compatible
Posted by Benjamin Mahler <bm...@apache.org>.
Tomek I think that patch also needs to address accepting strings for
floating point fields: float, double.
IMO, to avoid confusion, we want to get as close as possible to the proto3
specified mapping, which allows string inputs for all types of numeric
fields:
https://developers.google.com/protocol-buffers/docs/proto3#json
This means that on the output side, we need to figure out how to transition
to using strings for 64 bit integers without breaking our existing endpoint
consumers.
On Fri, Mar 24, 2017 at 12:31 PM, Anand Mazumdar <an...@apache.org> wrote:
> Hi Tomek,
>
> Looks like we dropped the ball on MESOS-5995
> (https://issues.apache.org/jira/browse/MESOS-5995). I assigned myself
> as the shepherd and would take a look next week.
>
> -anand
>
> On Thu, Mar 23, 2017 at 2:09 AM, Tomek Janiszewski <ja...@gmail.com>
> wrote:
> > I have a similar problem with protobuf and json. In my case numbers were
> > incorrectly unmarshaled. Here is an issue
> > https://issues.apache.org/jira/browse/MESOS-970 and review
> > https://reviews.apache.org/r/50851/
> >
> > czw., 23.03.2017, 09:54 użytkownik Olivier Sallou <
> olivier.sallou@irisa.fr>
> > napisał:
> >
> >> Hi,
> >>
> >> when transforming a protobug message to json with MessageToJson, the
> >> json is not compatible with the json format expected by Mesos master.
> >>
> >> For example, for volumes it generates
> >>
> >>
> >> volumes: [
> >>
> >> {'hostPath': '....',
> >>
> >> 'containerPath': '...',
> >>
> >> ...
> >>
> >> }
> >>
> >> ]
> >>
> >>
> >> but HTTP API expects "source" and "container_path"
> >>
> >> is it an expected behavior ? This prevents from "creating" a task in
> >> protobuf format and sending it to HTTP API with a protobug to json
> >> conversion.
> >>
> >> Thanks
> >>
> >> Olivier
> >>
> >> --
> >> Olivier Sallou
> >> IRISA / University of Rennes 1
> >> Campus de Beaulieu, 35000 RENNES - FRANCE
> >> Tel: 02.99.84.71.95
> >>
> >> gpg key id: 4096R/326D8438 (keyring.debian.org)
> >> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
> >>
> >>
>
Re: protbuf to json not compatible
Posted by Anand Mazumdar <an...@apache.org>.
Hi Tomek,
Looks like we dropped the ball on MESOS-5995
(https://issues.apache.org/jira/browse/MESOS-5995). I assigned myself
as the shepherd and would take a look next week.
-anand
On Thu, Mar 23, 2017 at 2:09 AM, Tomek Janiszewski <ja...@gmail.com> wrote:
> I have a similar problem with protobuf and json. In my case numbers were
> incorrectly unmarshaled. Here is an issue
> https://issues.apache.org/jira/browse/MESOS-970 and review
> https://reviews.apache.org/r/50851/
>
> czw., 23.03.2017, 09:54 użytkownik Olivier Sallou <ol...@irisa.fr>
> napisał:
>
>> Hi,
>>
>> when transforming a protobug message to json with MessageToJson, the
>> json is not compatible with the json format expected by Mesos master.
>>
>> For example, for volumes it generates
>>
>>
>> volumes: [
>>
>> {'hostPath': '....',
>>
>> 'containerPath': '...',
>>
>> ...
>>
>> }
>>
>> ]
>>
>>
>> but HTTP API expects "source" and "container_path"
>>
>> is it an expected behavior ? This prevents from "creating" a task in
>> protobuf format and sending it to HTTP API with a protobug to json
>> conversion.
>>
>> Thanks
>>
>> Olivier
>>
>> --
>> Olivier Sallou
>> IRISA / University of Rennes 1
>> Campus de Beaulieu, 35000 RENNES - FRANCE
>> Tel: 02.99.84.71.95
>>
>> gpg key id: 4096R/326D8438 (keyring.debian.org)
>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>>
>>
Re: protbuf to json not compatible
Posted by Tomek Janiszewski <ja...@gmail.com>.
I have a similar problem with protobuf and json. In my case numbers were
incorrectly unmarshaled. Here is an issue
https://issues.apache.org/jira/browse/MESOS-970 and review
https://reviews.apache.org/r/50851/
czw., 23.03.2017, 09:54 użytkownik Olivier Sallou <ol...@irisa.fr>
napisał:
> Hi,
>
> when transforming a protobug message to json with MessageToJson, the
> json is not compatible with the json format expected by Mesos master.
>
> For example, for volumes it generates
>
>
> volumes: [
>
> {'hostPath': '....',
>
> 'containerPath': '...',
>
> ...
>
> }
>
> ]
>
>
> but HTTP API expects "source" and "container_path"
>
> is it an expected behavior ? This prevents from "creating" a task in
> protobuf format and sending it to HTTP API with a protobug to json
> conversion.
>
> Thanks
>
> Olivier
>
> --
> Olivier Sallou
> IRISA / University of Rennes 1
> Campus de Beaulieu, 35000 RENNES - FRANCE
> Tel: 02.99.84.71.95
>
> gpg key id: 4096R/326D8438 (keyring.debian.org)
> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>
>