mesos git commit: MESOS-3566 Description of RecordIO format

[me...@apache.org]

Repository: mesos
Updated Branches:
  refs/heads/master 9c54387db -> 8e34c5218


MESOS-3566 Description of RecordIO format

Added the description of the RecordIO format to the HTTP API
document with example code (Python) to decode.

Review: https://reviews.apache.org/r/39452


Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/8e34c521
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/8e34c521
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/8e34c521

Branch: refs/heads/master
Commit: 8e34c5218f659f228f64b69656a6be3f374e690a
Parents: 9c54387
Author: Marco Massenzio <ma...@mesosphere.io>
Authored: Sun Nov 8 00:23:36 2015 -0800
Committer: Adam B <ad...@mesosphere.io>
Committed: Sun Nov 8 00:25:18 2015 -0800

----------------------------------------------------------------------
 docs/scheduler-http-api.md | 45 ++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 44 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/mesos/blob/8e34c521/docs/scheduler-http-api.md
----------------------------------------------------------------------
diff --git a/docs/scheduler-http-api.md b/docs/scheduler-http-api.md
index de6cfc9..38d3607 100644
--- a/docs/scheduler-http-api.md
+++ b/docs/scheduler-http-api.md
@@ -22,7 +22,50 @@ All the subsequent (non subscribe) requests to “/scheduler” endpoint (see de
 
 The following calls are currently accepted by the master. The canonical source of this information is [scheduler.proto](https://github.com/apache/mesos/blob/master/include/mesos/v1/scheduler/scheduler.proto) (NOTE: The protobuf definitions are subject to change before the beta API is finalized). Note that when sending JSON encoded Calls, schedulers should encode raw bytes in Base64 and strings in UTF-8.
 
-### SUBSCRIBE
+### RecordIO response format
+
+The response returned from the `SUBSCRIBE` call (see [below](#subscribe) is encoded in RecordIO format, which essentially prepends to a single record (either JSON or serialized Protobuf) its length in bytes, followed by a newline and then the data:
+
+The [BNF grammar](http://www.w3.org/Protocols/rfc2616/rfc2616-sec2.html#sec2.1) for a RecordIO-encoded streaming response is:
+
+```
+    records         = *record
+
+    record          = record-size LF record-data
+
+    record-size     = 1*DIGIT
+    record-data     = record-size(OCTET)
+```
+
+`record-size` should be interpreted as an unsigned 64-bit integer (`uint64`).
+
+For example, a stream may look like:
+
+```
+128\n
+{“type”: “SUBSCRIBED”,“subscribed”: {“framework_id”: {“value”:“12220-3440-12532-2345”},...}104\n
+{“framework_id”: {“value”: “12220-3440-12532-2345”},...{“value” : “12220-3440-12532-O12”},}208\n
+...
+```
+
+In pseudo-code, this could be parsed with something like the following:
+
+```
+  while (true) {
+    do {
+      lengthBytes = readline()
+    } while (lengthBytes.length < 1)
+
+    messageLength = parseInt(lengthBytes);
+    messageBytes = read(messageLength);
+    process(messageBytes);
+  }
+```
+
+Network intermediaries e.g. proxies are free to change the chunk boundaries and this should not have any effect on the recipient application (scheduler layer). We wanted a way to delimit/encode two events for JSON/Protobuf responses consistently and RecordIO format allowed us to do that.
+
+
+### <a id="subscribe"></a>SUBSCRIBE
 
 This is the first step in the communication process between the scheduler and the master. This is also to be considered as subscription to the “/scheduler” events stream.