[jira] [Comment Edited] (MESOS-3067) Implement a streaming response decoder for events stream

["Vinod Kone (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MESOS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15306291#comment-15306291 ] 

Vinod Kone edited comment on MESOS-3067 at 5/30/16 6:30 AM:
------------------------------------------------------------

We chose RecordIO format because it can be applied consistently to both JSON and serialized protobufs. As for the content type, agree that application/json is incorrect but that is what Twitter streaming API and other APIs I have looked around that does RecordIO streaming call it :( Maybe we should call it application/x-json-recordio and application/x-protobuf-recordio respectively?


was (Author: vinodkone):
We chose RecordIO format because it can be applied consistently to both JSON and serialized protobufs. As for the content type, agree that application/json is incorrect but that is what Twitter streaming API and other APIs I have looked around that does RecordIO streaming call it :( Maybe we should call it application/x-json-recordio?

> Implement a streaming response decoder for events stream
> --------------------------------------------------------
>
>                 Key: MESOS-3067
>                 URL: https://issues.apache.org/jira/browse/MESOS-3067
>             Project: Mesos
>          Issue Type: Task
>            Reporter: Anand Mazumdar
>            Assignee: Benjamin Mahler
>             Fix For: 0.24.0
>
>
> We need a streaming response decoder to de-serialize chunks sent from the master on the events stream.
> From the HTTP API design doc:
> Master encodes each Event in RecordIO format, i.e. a string representation of length of the event in bytes followed by JSON or binary Protobuf  (possibly compressed) encoded event.
> As of now for getting the basic features right , this is being done in the test-cases:
> {code}
>   auto reader = response.get().reader;
>   ASSERT_SOME(reader);
>   Future<std::string> eventFuture = reader.get().read();
>   AWAIT_READY(eventFuture);
>   Event event;
>   event.ParseFromString(eventFuture.get());
> {code}
> Two things need to happen:
> - We need master to emit events in RecordIO format i.e. event size followed by the serialized event instead of just the serialized events as is the case now.
> - The decoder class should then abstract away the logic of reading the response and de-serializing events from the stream.
> Ideally, the decoder should work with both "json" and "protobuf" responses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)