You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mesos.apache.org by al...@apache.org on 2018/06/14 13:35:16 UTC
[1/2] mesos git commit: Moved RecordIO specification into separate
file.
Repository: mesos
Updated Branches:
refs/heads/master f8c8c35af -> e25385a2f
Moved RecordIO specification into separate file.
Created a new file 'docs/recordio.md' that expands upon the
description of the RecordIO data format which was previously
described in 'scheduler-http-api.md'.
Review: https://reviews.apache.org/r/67546/
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/ee14ed11
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/ee14ed11
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/ee14ed11
Branch: refs/heads/master
Commit: ee14ed11fb307e652a32f1b421ab15795bd6160d
Parents: f8c8c35
Author: Benno Evers <be...@mesosphere.com>
Authored: Thu Jun 14 15:34:32 2018 +0200
Committer: Alexander Rukletsov <al...@apache.org>
Committed: Thu Jun 14 15:34:32 2018 +0200
----------------------------------------------------------------------
docs/home.md | 1 +
docs/operator-http-api.md | 4 +-
docs/recordio.md | 88 +++++++++++++++++++++++++++++++++++++++++
docs/scheduler-http-api.md | 41 +------------------
4 files changed, 92 insertions(+), 42 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/mesos/blob/ee14ed11/docs/home.md
----------------------------------------------------------------------
diff --git a/docs/home.md b/docs/home.md
index adefc4d..077096d 100644
--- a/docs/home.md
+++ b/docs/home.md
@@ -71,6 +71,7 @@ layout: documentation
* [HTTP Endpoints](endpoints/) for available HTTP endpoints.
* [API Client Libraries](api-client-libraries.md) lists client libraries for the HTTP APIs.
* [API Versioning](versioning.md) describes HTTP API and release versioning.
+* [RecordIO](recordio.md) describes the RecordIO format used by the streaming endpoints of the HTTP API.
* Writing Frameworks
* [Framework Development Guide](app-framework-development-guide.md) describes how to build applications on top of Mesos.
* [Guide for Designing Highly Available Mesos Frameworks](high-availability-framework-guide.md)
http://git-wip-us.apache.org/repos/asf/mesos/blob/ee14ed11/docs/operator-http-api.md
----------------------------------------------------------------------
diff --git a/docs/operator-http-api.md b/docs/operator-http-api.md
index 4acbee9..9a37dc5 100644
--- a/docs/operator-http-api.md
+++ b/docs/operator-http-api.md
@@ -13,7 +13,7 @@ Similar to the [Scheduler](scheduler-http-api.md) and [Executor](executor-http-a
For requests that Mesos can answer synchronously and immediately, an HTTP response will be sent with status **200 OK**, possibly including a response body encoded in JSON or Protobuf. The encoding depends on the **Accept** header present in the request (the default encoding is JSON). Responses will be gzip compressed if the **Accept-Encoding** header is set to "gzip".
-For requests that require asynchronous processing (e.g., `RESERVE_RESOURCES`), an HTTP response will be sent with status **202 Accepted**. For requests that result in a stream of events (`SUBSCRIBE`), a streaming HTTP response with [RecordIO](scheduler-http-api.md#recordio-response-format) encoding is sent. Currently, gzip compression is not supported for streaming responses.
+For requests that require asynchronous processing (e.g., `RESERVE_RESOURCES`), an HTTP response will be sent with status **202 Accepted**. For requests that result in a stream of events (`SUBSCRIBE`), a streaming HTTP response with [RecordIO](recordio.md) encoding is sent. Currently, gzip compression is not supported for streaming responses.
## Master API
@@ -2504,7 +2504,7 @@ Transfer-Encoding: chunked
<more events>
```
-The client is expected to keep a **persistent** connection open to the endpoint even after getting a `SUBSCRIBED` HTTP Response event. This is indicated by "Connection: keep-alive" and "Transfer-Encoding: chunked" headers with *no* "Content-Length" header set. All subsequent events generated by Mesos are streamed on this connection. The master encodes each Event in [RecordIO](scheduler-http-api.md#recordio-response-format) format, i.e., string representation of length of the event in bytes followed by JSON or binary Protobuf encoded event.
+The client is expected to keep a **persistent** connection open to the endpoint even after getting a `SUBSCRIBED` HTTP Response event. This is indicated by "Connection: keep-alive" and "Transfer-Encoding: chunked" headers with *no* "Content-Length" header set. All subsequent events generated by Mesos are streamed on this connection. The master encodes each Event in [RecordIO](recordio.md) format, i.e., string representation of length of the event in bytes followed by JSON or binary Protobuf encoded event.
The following events are currently sent by the master. The canonical source of this information is at [master.proto](https://github.com/apache/mesos/blob/master/include/mesos/v1/master/master.proto). Note that when sending JSON encoded events, master encodes raw bytes in Base64 and strings in UTF-8.
http://git-wip-us.apache.org/repos/asf/mesos/blob/ee14ed11/docs/recordio.md
----------------------------------------------------------------------
diff --git a/docs/recordio.md b/docs/recordio.md
new file mode 100644
index 0000000..f732793
--- /dev/null
+++ b/docs/recordio.md
@@ -0,0 +1,88 @@
+---
+title: Apache Mesos - RecordIO Data Format
+layout: documentation
+---
+
+# RecordIO Data Format
+
+## Overview
+
+RecordIO is the name for a set of binary data exchange formats.
+The basic idea is to divide the data into individual chunks, called 'records',
+and then to prepend to every record its length in bytes, followed by the data.
+
+Since there is no formal specification of the RecordIO format, there tend
+to be slight incompatibilities between RecordIO implementations. Common
+differences include, for example, a magic value at the beginning of the
+stream to indicate the stream format, encoding the record length using
+fixed-size fields with native integers or extra fields to suppord compressed
+record content.
+
+Therefore, when using a third-party library, one needs to verify that
+the implementation matches the RecordIO format used by Mesos, as described
+below.
+
+
+## Mesos RecordIO Format
+
+The [BNF grammar](http://www.w3.org/Protocols/rfc2616/rfc2616-sec2.html#sec2.1)
+for a RecordIO-encoded streaming response is:
+
+```
+ records = *record
+
+ record = record-size LF record-data
+
+ record-size = 1*DIGIT
+ record-data = record-size(OCTET)
+```
+
+`record-size` should be interpreted as an unsigned 64-bit integer (`uint64`).
+
+For example, a stream may look like:
+
+```
+128\n
+{"type": "SUBSCRIBED","subscribed": {"framework_id": {"value":"12220-3440-12532-2345"},"heartbeat_interval_seconds":15.0}20\n
+{"type":"HEARTBEAT"}675\n
+...
+```
+
+In pseudo-code, this could be parsed with something like the following:
+
+```
+ while (true) {
+ do {
+ lengthBytes = readline()
+ } while (lengthBytes.length < 1)
+
+ messageLength = parseInt(lengthBytes);
+ messageBytes = read(messageLength);
+ process(messageBytes);
+ }
+```
+
+Network intermediaries (e.g., proxies) are free to change the chunk
+boundaries; this should not have any effect on the recipient application.
+
+
+## Streaming HTTP requests
+
+Within Mesos, RecordIO is used as data format for API endpoints
+that support streaming responses. The implementation expects the
+client to be aware of the following non-standard HTTP header fields
+when using such endpoints:
+
+The `Message-Accept` header must be set by clients that request a
+streaming response from the server by setting the `Accept` header to
+`application/recordio`. It specifies which media types the client
+accepts encoded in the RecordIO stream, and its content must have the
+same format as the standard HTTP `Accept` header.
+
+The `Message-Content-Type` is set by the server when streaming the
+response to an API call. It specifies the content type of the data
+encoded in the RecordIO stream and has the same format as the
+standard HTTP `Content-Type` header.
+
+Currently, the only supported MIME types for both header fields are
+`application/json` and `application/x-protobuf`.
http://git-wip-us.apache.org/repos/asf/mesos/blob/ee14ed11/docs/scheduler-http-api.md
----------------------------------------------------------------------
diff --git a/docs/scheduler-http-api.md b/docs/scheduler-http-api.md
index bea8817..54d248e 100644
--- a/docs/scheduler-http-api.md
+++ b/docs/scheduler-http-api.md
@@ -48,46 +48,7 @@ The following calls are currently accepted by the master. The canonical source o
<a id="recordio-response-format"></a>
### RecordIO response format
-The response returned from the `SUBSCRIBE` call (see [below](#subscribe)) is encoded in RecordIO format, which essentially prepends to a single record (either JSON or serialized Protobuf) its length in bytes, followed by a newline and then the data:
-
-The [BNF grammar](http://www.w3.org/Protocols/rfc2616/rfc2616-sec2.html#sec2.1) for a RecordIO-encoded streaming response is:
-
-```
- records = *record
-
- record = record-size LF record-data
-
- record-size = 1*DIGIT
- record-data = record-size(OCTET)
-```
-
-`record-size` should be interpreted as an unsigned 64-bit integer (`uint64`).
-
-For example, a stream may look like:
-
-```
-128\n
-{"type": "SUBSCRIBED","subscribed": {"framework_id": {"value":"12220-3440-12532-2345"},"heartbeat_interval_seconds":15.0}20\n
-{"type":"HEARTBEAT"}675\n
-...
-```
-
-In pseudo-code, this could be parsed with something like the following:
-
-```
- while (true) {
- do {
- lengthBytes = readline()
- } while (lengthBytes.length < 1)
-
- messageLength = parseInt(lengthBytes);
- messageBytes = read(messageLength);
- process(messageBytes);
- }
-```
-
-Network intermediaries (e.g., proxies) are free to change the chunk boundaries; this should not have any effect on the recipient application (scheduler). We wanted a way to delimit/encode two events for JSON/Protobuf responses consistently and RecordIO format allowed us to do that.
-
+The response returned from the `SUBSCRIBE` call (see [below](#subscribe)) is encoded in RecordIO format, which essentially prepends to a single record (either JSON or serialized Protobuf) its length in bytes, followed by a newline and then the data. See [RecordIO Format](recordio.md) for details.
<a id="subscribe"></a>
### SUBSCRIBE
[2/2] mesos git commit: Added check for API endpoints not supporting
streaming responses.
Posted by al...@apache.org.
Added check for API endpoints not supporting streaming responses.
A client can request any endpoint to stream its response using
the RecordIO format. However, we never checked whether the selected
endpoint actually supports streaming, leading to an agent crash
when it was used for endpoints that did not expect such a request.
Review: https://reviews.apache.org/r/67547/
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/e25385a2
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/e25385a2
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/e25385a2
Branch: refs/heads/master
Commit: e25385a2f9337194164786d60a5e107140a78ae6
Parents: ee14ed1
Author: Benno Evers <be...@mesosphere.com>
Authored: Thu Jun 14 15:34:42 2018 +0200
Committer: Alexander Rukletsov <al...@apache.org>
Committed: Thu Jun 14 15:34:42 2018 +0200
----------------------------------------------------------------------
src/slave/http.cpp | 15 ++++++++++++---
src/tests/api_tests.cpp | 21 +++++++++++++++++++++
2 files changed, 33 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/mesos/blob/e25385a2/src/slave/http.cpp
----------------------------------------------------------------------
diff --git a/src/slave/http.cpp b/src/slave/http.cpp
index a6739e1..aa9cdc5 100644
--- a/src/slave/http.cpp
+++ b/src/slave/http.cpp
@@ -466,6 +466,8 @@ Future<Response> Http::api(
Option<ContentType> messageAcceptType;
if (streamingMediaType(acceptType)) {
+ // Note that `acceptsMediaType()` returns true if the given headers
+ // field does not exist, i.e. by default we return JSON here.
if (request.acceptsMediaType(MESSAGE_ACCEPT, APPLICATION_JSON)) {
messageAcceptType = ContentType::JSON;
} else if (request.acceptsMediaType(MESSAGE_ACCEPT, APPLICATION_PROTOBUF)) {
@@ -541,8 +543,8 @@ Future<Response> Http::_api(
if (streamingMediaType(mediaTypes.content) &&
call.type() != mesos::agent::Call::ATTACH_CONTAINER_INPUT) {
return UnsupportedMediaType(
- "Streaming 'Content-Type' " + stringify(mediaTypes.content) + " is not "
- "supported for " + stringify(call.type()) + " call");
+ "Streaming 'Content-Type' " + stringify(mediaTypes.content) + " is "
+ "not supported for " + stringify(call.type()) + " call");
} else if (!streamingMediaType(mediaTypes.content) &&
call.type() == mesos::agent::Call::ATTACH_CONTAINER_INPUT) {
return UnsupportedMediaType(
@@ -550,6 +552,13 @@ Future<Response> Http::_api(
" for " + stringify(call.type()) + " call");
}
+ if (streamingMediaType(mediaTypes.accept) &&
+ call.type() != mesos::agent::Call::ATTACH_CONTAINER_OUTPUT &&
+ call.type() != mesos::agent::Call::LAUNCH_NESTED_CONTAINER_SESSION) {
+ return NotAcceptable("Streaming response is not supported for " +
+ stringify(call.type()) + " call");
+ }
+
// Each handler must log separately to add context
// it might extract from the nested message.
switch (call.type()) {
@@ -2480,7 +2489,7 @@ Future<Response> Http::_launchContainer(
const Option<Resources>& resources,
const Option<ContainerInfo>& containerInfo,
const Option<ContainerClass>& containerClass,
- ContentType acceptType,
+ ContentType,
const Owned<ObjectApprovers>& approvers) const
{
Option<string> user;
http://git-wip-us.apache.org/repos/asf/mesos/blob/e25385a2/src/tests/api_tests.cpp
----------------------------------------------------------------------
diff --git a/src/tests/api_tests.cpp b/src/tests/api_tests.cpp
index 15dbbc1..85635a8 100644
--- a/src/tests/api_tests.cpp
+++ b/src/tests/api_tests.cpp
@@ -6897,6 +6897,27 @@ TEST_F(AgentAPITest, HeaderValidation)
AWAIT_EXPECT_RESPONSE_STATUS_EQ(http::UnsupportedMediaType().status,
response);
}
+
+ // Setting 'Accept: application/recordio' header for a non-streaming request.
+ {
+ v1::agent::Call call;
+ call.set_type(v1::agent::Call::GET_CONTAINERS);
+
+ ContentType contentType = ContentType::JSON;
+
+ http::Headers headers = createBasicAuthHeaders(DEFAULT_CREDENTIAL);
+ headers["Accept"] = "application/recordio";
+
+ Future<http::Response> response = http::post(
+ slave.get()->pid,
+ "api/v1",
+ headers,
+ serialize(contentType, call),
+ stringify(contentType));
+
+ AWAIT_EXPECT_RESPONSE_STATUS_EQ(http::NotAcceptable().status,
+ response);
+ }
}