You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mesos.apache.org by ka...@apache.org on 2016/02/01 03:49:28 UTC
svn commit: r1727886 [16/16] - in /mesos/site: ./ publish/
publish/assets/img/documentation/ publish/community/ publish/documentation/
publish/documentation/allocation-module/
publish/documentation/app-framework-development-guide/
publish/documentation...
Added: mesos/site/source/documentation/latest/sandbox.md
URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/sandbox.md?rev=1727886&view=auto
==============================================================================
--- mesos/site/source/documentation/latest/sandbox.md (added)
+++ mesos/site/source/documentation/latest/sandbox.md Mon Feb 1 02:49:25 2016
@@ -0,0 +1,172 @@
+---
+layout: documentation
+---
+
+# Mesos "Sandbox"
+
+Mesos refers to the "sandbox" as a temporary directory that holds files specific
+to a single executor. Each time an executor is run, the executor is given its
+own sandbox and the executor's working directory is set to the sandbox.
+
+## Sandbox files
+
+The sandbox holds:
+
+* Files [fetched by Mesos](/documentation/latest/fetcher/), prior to starting
+ the executor's tasks.
+* The output of the executor and tasks (as files "stdout" and "stderr").
+* Files created by the executor and tasks, with some exceptions.
+
+> **NOTE:** With the introduction of
+> [persistent volumes](/documentation/latest/persistent-volume/), executors
+> and tasks should never create files outside of the sandbox. However,
+> some containerizers do not enforce this sandboxing.
+
+## <a name="where-is-it"></a>Where is the sandbox?
+
+The sandbox is located within the agent's working directory (which is specified
+via the `--work_dir` flag). To find a particular executor's sandbox, you must
+know the agent's ID, the executor's framework's ID, and the executor's ID.
+Each run of the executor will have a corresponding sandbox, denoted by a
+container ID.
+
+The sandbox is located on the agent, inside a directory tree like the following:
+
+```
+root ('--work_dir')
+|-- slaves
+| |-- latest (symlink)
+| |-- <agent ID>
+| |-- frameworks
+| |-- <framework ID>
+| |-- executors
+| |-- <executor ID>
+| |-- runs
+| |-- latest (symlink)
+| |-- <container ID> (Sandbox!)
+```
+
+## Using the sandbox
+
+> **NOTE:** For anything other than Mesos, the executor, or the task(s), the
+> sandbox should be considered a read-only directory. This is not enforced via
+> permissions, but the executor/tasks may malfunction if the sandbox is
+> mutated unexpectedly.
+
+### Via a file browser
+
+If you have access to the machine running the agent, you can [navigate to the
+sandbox directory directly](#where-is-it).
+
+### Via the Mesos web UI
+
+Sandboxes can be browsed and downloaded via the Mesos web UI. Tasks and
+executors will be shown with a "Sandbox" link. Any files that live in the
+sandbox will appear in the web UI.
+
+### Via the `/files` endpoint
+
+Underneath the web UI, the files are fetched from the agent via the `/files`
+endpoint running on the agent.
+
+<table class="table table-striped">
+ <thead>
+ <tr>
+ <th width="30%">
+ Endpoint
+ </th>
+ <th>
+ Description
+ </th>
+ </tr>
+ </thead>
+
+ <tr>
+ <td>
+ <code>/files/browse?path=...</code>
+ </td>
+ <td>
+ Returns a JSON list of files and directories contained in the path.
+ Each list is a JSON object containing all the fields normally found in
+ <code>ls -l</code>.
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <code>/files/debug</code>
+ </td>
+ <td>
+ Returns a JSON object holding the internal mapping of files managed by
+ this endpoint. This endpoint can be used to quickly fetch the paths
+ of all files exposed on the agent.
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <code>/files/download?path=...</code>
+ </td>
+ <td>
+ Returns the raw contents of the file located at the given path.
+ Where the file extension is understood, the <code>Content-Type</code>
+ header will be set appropriately.
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <code>/files/read?path=...</code>
+ </td>
+ <td>
+ Reads a chunk of the file located at the given path and returns a JSON
+ object containing the read <code>"data"</code> and the
+ <code>"offset"</code> in bytes.
+ <blockquote>
+ <p>
+ <strong>NOTE:</strong> This endpoint is not designed to read
+ arbitrary binary files. Binary files may be returned as
+ invalid/un-parseable JSON.
+ Use <code>/files/download</code> instead.
+ </p>
+ </blockquote>
+ Optional query parameters:
+ <ul>
+ <li><code>offset</code> - can be used to page through the file.</li>
+ <li><code>length</code> - maximum size of the chunk to read.</li>
+ </ul>
+ </td>
+ </tr>
+</table>
+
+## Sandbox size
+
+The maximum size of the sandbox is dependent on the containerization of the
+executor and isolators :
+
+* Mesos containerizer - For backwards compatibility, the Mesos containerizer
+ does not enforce a container's disk quota by default. However, if the
+ `--enforce_container_disk_quota` flag is enabled on the agent, and
+ `posix/disk` is specified in the `--isolation` flag, the executor
+ will be killed if the sandbox size exceeds the executor's `disk` resource.
+* Docker containerizer - As of Docker `1.9.1`, the Docker containerizer
+ does not enforce nor support a disk quota. See the
+ [Docker issue](https://github.com/docker/docker/issues/3804).
+* [External containerizer](/documentation/latest/external-containerizer/).
+
+## Sandbox lifecycle
+
+Sandbox files are scheduled for garbage collection when:
+
+* An executor is removed or terminated.
+* A framework is removed.
+* An executor is recovered unsuccessfully during agent recovery.
+
+> **NOTE:** During agent recovery, all of the executor's runs, except for the
+> latest run, are scheduled for garbage collection as well.
+
+Garbage collection is scheduled based on the `--gc_delay` agent flag. By
+default, this is one week since the sandbox was last modified.
+After the delay, the files are deleted.
+
+Additionally, according to the `--disk_watch_interval` agent flag, files
+scheduled for garbage collection are pruned based on the available disk and
+the `--gc_disk_headroom` agent flag.
+See [the formula here](configuration.md#gc_disk_headroom).
Modified: mesos/site/source/documentation/latest/scheduler-http-api.md
URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/scheduler-http-api.md?rev=1727886&r1=1727885&r2=1727886&view=diff
==============================================================================
--- mesos/site/source/documentation/latest/scheduler-http-api.md (original)
+++ mesos/site/source/documentation/latest/scheduler-http-api.md Mon Feb 1 02:49:25 2016
@@ -9,19 +9,21 @@ Mesos 0.24.0 added **experimental** supp
## Overview
-The scheduler interacts with Mesos via â/api/v1/schedulerâ endpoint hosted by the Mesos master. The fully qualified URL of the endpoint might look like:
+The scheduler interacts with Mesos via "/api/v1/scheduler" endpoint hosted by the Mesos master. The fully qualified URL of the endpoint might look like:
http://masterhost:5050/api/v1/scheduler
-Note that we refer to this endpoint with its suffix "/scheduler" in the rest of this document. This endpoint accepts HTTP POST requests with data encoded as JSON (Content-Type: application/json) or binary Protobuf (Content-Type: application/x-protobuf). The first request that a scheduler sends to â/schedulerâ endpoint is called SUBSCRIBE and results in a streaming response (â200 OKâ status code with Transfer-Encoding: chunked). **Schedulers are expected to keep the subscription connection open as long as possible (barring errors in network, software, hardware etc.) and incrementally process the response** (NOTE: HTTP client libraries that can only parse the response after the connection is closed cannot be used). For the encoding used, please refer to **Events** section below.
+Note that we refer to this endpoint with its suffix "/scheduler" in the rest of this document. This endpoint accepts HTTP POST requests with data encoded as JSON (Content-Type: application/json) or binary Protobuf (Content-Type: application/x-protobuf). The first request that a scheduler sends to "/scheduler" endpoint is called SUBSCRIBE and results in a streaming response ("200 OK" status code with Transfer-Encoding: chunked). **Schedulers are expected to keep the subscription connection open as long as possible (barring errors in network, software, hardware etc.) and incrementally process the response** (NOTE: HTTP client libraries that can only parse the response after the connection is closed cannot be used). For the encoding used, please refer to **Events** section below.
-All the subsequent (non subscribe) requests to â/schedulerâ endpoint (see details below in **Calls** section) must be sent using a different connection(s) than the one being used for subscription. Master responds to these HTTP POST requests with â202 Acceptedâ status codes (or, for unsuccessful requests, with 4xx or 5xx status codes; details in later sections). The â202 Acceptedâ response means that a request has been accepted for processing, not that the processing of the request has been completed. The request might or might not be acted upon by Mesos (e.g., master fails during the processing of the request). Any asynchronous responses from these requests will be streamed on long-lived subscription connection.
+All the subsequent (non subscribe) requests to "/scheduler" endpoint (see details below in **Calls** section) must be sent using a different connection(s) than the one being used for subscription. Master responds to these HTTP POST requests with "202 Accepted" status codes (or, for unsuccessful requests, with 4xx or 5xx status codes; details in later sections). The "202 Accepted" response means that a request has been accepted for processing, not that the processing of the request has been completed. The request might or might not be acted upon by Mesos (e.g., master fails during the processing of the request). Any asynchronous responses from these requests will be streamed on long-lived subscription connection.
## Calls
The following calls are currently accepted by the master. The canonical source of this information is [scheduler.proto](https://github.com/apache/mesos/blob/master/include/mesos/v1/scheduler/scheduler.proto) (NOTE: The protobuf definitions are subject to change before the beta API is finalized). Note that when sending JSON encoded Calls, schedulers should encode raw bytes in Base64 and strings in UTF-8.
+
+<a id="recordio-response-format"></a>
### RecordIO response format
The response returned from the `SUBSCRIBE` call (see [below](#subscribe) is encoded in RecordIO format, which essentially prepends to a single record (either JSON or serialized Protobuf) its length in bytes, followed by a newline and then the data:
@@ -43,8 +45,8 @@ For example, a stream may look like:
```
128\n
-{âtypeâ: âSUBSCRIBEDâ,âsubscribedâ: {âframework_idâ: {âvalueâ:â12220-3440-12532-2345â},...}104\n
-{âframework_idâ: {âvalueâ: â12220-3440-12532-2345â},...{âvalueâ : â12220-3440-12532-O12â},}208\n
+{"type": "SUBSCRIBED","subscribed": {"framework_id": {"value":"12220-3440-12532-2345"},...}104\n
+{"framework_id": {"value": "12220-3440-12532-2345"},...{"value" : "12220-3440-12532-O12"},}208\n
...
```
@@ -65,9 +67,10 @@ In pseudo-code, this could be parsed wit
Network intermediaries e.g. proxies are free to change the chunk boundaries and this should not have any effect on the recipient application (scheduler layer). We wanted a way to delimit/encode two events for JSON/Protobuf responses consistently and RecordIO format allowed us to do that.
-### <a id="subscribe"></a>SUBSCRIBE
+<a id="subscribe"></a>
+### SUBSCRIBE
-This is the first step in the communication process between the scheduler and the master. This is also to be considered as subscription to the â/schedulerâ events stream.
+This is the first step in the communication process between the scheduler and the master. This is also to be considered as subscription to the "/scheduler" events stream.
To subscribe with the master, the scheduler sends a HTTP POST request with encoded `SUBSCRIBE` message with the required FrameworkInfo. Note that if "subscribe.framework_info.id" is not set, master considers the scheduler as a new one and subscribes it by assigning it a FrameworkID. The HTTP response is a stream with RecordIO encoding, with the first event being `SUBSCRIBED` event (see details in **Events** section).
@@ -82,15 +85,15 @@ Accept: application/json
Connection: close
{
- âtypeâ : âSUBSCRIBEâ,
+ "type" : "SUBSCRIBE",
- âsubscribeâ : {
- âframework_infoâ : {
- âuserâ : âfooâ,
- ânameâ : âExample HTTP Frameworkâ
+ "subscribe" : {
+ "framework_info" : {
+ "user" : "foo",
+ "name" : "Example HTTP Framework"
},
- âforceâ : true
+ "force" : true
}
}
@@ -102,22 +105,22 @@ Transfer-Encoding: chunked
<event length>
{
- âtypeâ : âSUBSCRIBEDâ,
- âsubscribedâ : {
- âframework_idâ : {âvalueâ:â12220-3440-12532-2345â},
+ "type" : "SUBSCRIBED",
+ "subscribed" : {
+ "framework_id" : {"value":"12220-3440-12532-2345"},
"heartbeat_interval_seconds" : 15
}
}
<more events>
```
-Alternatively, if âsubscribe.framework_info.idâ is set, master considers this a request from an already subscribed scheduler reconnecting after a disconnection (e.g., due to failover or network disconnection) and responds with `SUBSCRIBED` event containing the same FrameworkID. The âsubscribe.forceâ field describes how the master reacts when multiple scheduler instances (with the same framework id) attempt to subscribe with the master at the same time (e.g., due to network partition). See the semantics in **Disconnections** section below.
+Alternatively, if "subscribe.framework_info.id" is set, master considers this a request from an already subscribed scheduler reconnecting after a disconnection (e.g., due to failover or network disconnection) and responds with `SUBSCRIBED` event containing the same FrameworkID. The "subscribe.force" field describes how the master reacts when multiple scheduler instances (with the same framework id) attempt to subscribe with the master at the same time (e.g., due to network partition). See the semantics in **Disconnections** section below.
-NOTE: In the old version of the API, (re-)registered callbacks also included MasterInfo, which contained information about the master the driver currently connected to. With the new API, since schedulers explicitly subscribe with the leading master (see details below in **Master Detection** section), itâs not relevant anymore.
+NOTE: In the old version of the API, (re-)registered callbacks also included MasterInfo, which contained information about the master the driver currently connected to. With the new API, since schedulers explicitly subscribe with the leading master (see details below in **Master Detection** section), it's not relevant anymore.
If subscription fails for whatever reason (e.g., invalid request), a HTTP 4xx response is returned with the error message as part of the body and the connection is closed.
-Scheduler must make additional HTTP requests to the â/schedulerâ endpoint only after it has opened a persistent connection to it by sending a `SUBSCRIBE` request and received a `SUBSCRIBED` response. Calls made without subscription will result in a â403 Forbiddenâ instead of a â202 Acceptedâ response. A scheduler might also receive a â400 Bad Requestâ response if the HTTP request is malformed (e.g., malformed HTTP headers).
+Scheduler must make additional HTTP requests to the "/scheduler" endpoint only after it has opened a persistent connection to it by sending a `SUBSCRIBE` request and received a `SUBSCRIBED` response. Calls made without subscription will result in a "403 Forbidden" instead of a "202 Accepted" response. A scheduler might also receive a "400 Bad Request" response if the HTTP request is malformed (e.g., malformed HTTP headers).
### TEARDOWN
Sent by the scheduler when it wants to tear itself down. When Mesos receives this request it will shut down all executors (and consequently kill tasks) and remove persistent volumes (if requested). It then removes the framework and closes all open connections from this scheduler to the Master.
@@ -130,8 +133,8 @@ Host: masterhost:5050
Content-Type: application/json
{
- âframework_idâ : {âvalueâ : â12220-3440-12532-2345â},
- âtypeâ : âTEARDOWNâ,
+ "framework_id" : {"value" : "12220-3440-12532-2345"},
+ "type" : "TEARDOWN",
}
TEARDOWN Response:
@@ -139,7 +142,7 @@ HTTP/1.1 202 Accepted
```
### ACCEPT
-Sent by the scheduler when it accepts offer(s) sent by the master. The `ACCEPT` request includes the type of operations (e.g., launch task, reserve resources, create volumes) that the scheduler wants to perform on the offers. Note that until the scheduler replies (accepts or declines) to an offer, its resources are considered allocated to the framework. Also, any of the offerâs resources not used in the `ACCEPT` call (e.g., to launch a task) are considered declined and might be reoffered to other frameworks. In other words, the same `OfferID` cannot be used in more than one `ACCEPT` call. These semantics might change when we add new features to Mesos (e.g., persistence, reservations, optimistic offers, resizeTask, etc.).
+Sent by the scheduler when it accepts offer(s) sent by the master. The `ACCEPT` request includes the type of operations (e.g., launch task, reserve resources, create volumes) that the scheduler wants to perform on the offers. Note that until the scheduler replies (accepts or declines) to an offer, its resources are considered allocated to the framework. Also, any of the offer's resources not used in the `ACCEPT` call (e.g., to launch a task) are considered declined and might be reoffered to other frameworks. In other words, the same `OfferID` cannot be used in more than one `ACCEPT` call. These semantics might change when we add new features to Mesos (e.g., persistence, reservations, optimistic offers, resizeTask, etc.).
```
ACCEPT Request (JSON):
@@ -149,15 +152,15 @@ Host: masterhost:5050
Content-Type: application/json
{
- âframework_idâ : {âvalueâ : â12220-3440-12532-2345â},
- âtypeâ : âACCEPTâ,
- âacceptâ : {
- âoffer_idsâ : [
- {âvalueâ : â12220-3440-12532-O12â},
- {âvalueâ : â12220-3440-12532-O12â}
+ "framework_id" : {"value" : "12220-3440-12532-2345"},
+ "type" : "ACCEPT",
+ "accept" : {
+ "offer_ids" : [
+ {"value" : "12220-3440-12532-O12"},
+ {"value" : "12220-3440-12532-O12"}
],
- âoperationsâ : [ {âtypeâ : âLAUNCHâ, âlaunchâ : {...}} ],
- âfiltersâ : {...}
+ "operations" : [ {"type" : "LAUNCH", "launch" : {...}} ],
+ "filters" : {...}
}
}
@@ -177,14 +180,14 @@ Host: masterhost:5050
Content-Type: application/json
{
- âframework_idâ : {âvalueâ : â12220-3440-12532-2345â},
- âtypeâ : âDECLINEâ,
- âdeclineâ : {
- âoffer_idsâ : [
- {âvalueâ : â12220-3440-12532-O12â},
- {âvalueâ : â12220-3440-12532-O13â}
+ "framework_id" : {"value" : "12220-3440-12532-2345"},
+ "type" : "DECLINE",
+ "decline" : {
+ "offer_ids" : [
+ {"value" : "12220-3440-12532-O12"},
+ {"value" : "12220-3440-12532-O13"}
],
- âfiltersâ : {...}
+ "filters" : {...}
}
}
@@ -204,8 +207,8 @@ Host: masterhost:5050
Content-Type: application/json
{
- âframework_idâ : {âvalueâ : â12220-3440-12532-2345â},
- âtypeâ : âREVIVEâ,
+ "framework_id" : {"value" : "12220-3440-12532-2345"},
+ "type" : "REVIVE",
}
REVIVE Response:
@@ -224,11 +227,11 @@ Host: masterhost:5050
Content-Type: application/json
{
- âframework_idâ : {âvalueâ : â12220-3440-12532-2345â},
- âtypeâ : âKILLâ,
- âkillâ : {
- âtask_idâ : {âvalueâ : â12220-3440-12532-my-taskâ},
- âagent_idâ : {âvalueâ : â12220-3440-12532-S1233â}
+ "framework_id" : {"value" : "12220-3440-12532-2345"},
+ "type" : "KILL",
+ "kill" : {
+ "task_id" : {"value" : "12220-3440-12532-my-task"},
+ "agent_id" : {"value" : "12220-3440-12532-S1233"}
}
}
@@ -238,7 +241,7 @@ HTTP/1.1 202 Accepted
```
### SHUTDOWN
-Sent by the scheduler to shutdown a specific custom executor (NOTE: This is a new call that was not present in the old API). When an executor gets a shutdown event, it is expected to kill all its tasks (and send `TASK_KILLED` updates) and terminate. If an executor doesnât terminate within a certain timeout (configurable via â--executor_shutdown_grace_periodâ agent flag), the agent will forcefully destroy the container (executor and its tasks) and transitions its active tasks to `TASK_LOST`.
+Sent by the scheduler to shutdown a specific custom executor (NOTE: This is a new call that was not present in the old API). When an executor gets a shutdown event, it is expected to kill all its tasks (and send `TASK_KILLED` updates) and terminate. If an executor doesn't terminate within a certain timeout (configurable via "--executor_shutdown_grace_period" agent flag), the agent will forcefully destroy the container (executor and its tasks) and transitions its active tasks to `TASK_LOST`.
```
SHUTDOWN Request (JSON):
@@ -248,11 +251,11 @@ Host: masterhost:5050
Content-Type: application/json
{
- âframework_idâ : {âvalueâ : â12220-3440-12532-2345â},
- âtypeâ : âSHUTDOWNâ,
- âshutdownâ : {
- âexecutor_idâ : {âvalueâ : â123450-2340-1232-my-executorâ},
- âagent_idâ : {âvalueâ : â12220-3440-12532-S1233â}
+ "framework_id" : {"value" : "12220-3440-12532-2345"},
+ "type" : "SHUTDOWN",
+ "shutdown" : {
+ "executor_id" : {"value" : "123450-2340-1232-my-executor"},
+ "agent_id" : {"value" : "12220-3440-12532-S1233"}
}
}
@@ -262,7 +265,7 @@ HTTP/1.1 202 Accepted
```
### ACKNOWLEDGE
-Sent by the scheduler to acknowledge a status update. Note that with the new API, schedulers are responsible for explicitly acknowledging the receipt of status updates that have âstatus.uuid()â set. These status updates are reliably retried until they are acknowledged by the scheduler. The scheduler must not acknowledge status updates that do not have âstatus.uuid()â set as they are not retried. "uuid" is raw bytes encoded in Base64.
+Sent by the scheduler to acknowledge a status update. Note that with the new API, schedulers are responsible for explicitly acknowledging the receipt of status updates that have "status.uuid()" set. These status updates are reliably retried until they are acknowledged by the scheduler. The scheduler must not acknowledge status updates that do not have "status.uuid()" set as they are not retried. "uuid" is raw bytes encoded in Base64.
```
ACKNOWLEDGE Request (JSON):
@@ -272,12 +275,12 @@ Host: masterhost:5050
Content-Type: application/json
{
- âframework_idâ : {âvalueâ : â12220-3440-12532-2345â},
- âtypeâ : âACKNOWLEDGEâ,
- âacknowledgeâ : {
- âagent_idâ : {âvalueâ : â12220-3440-12532-S1233â},
- âtask_idâ : {âvalueâ : â12220-3440-12532-my-taskâ},
- âuuidâ : âjhadf73jhakdlfha723adfâ
+ "framework_id" : {"value" : "12220-3440-12532-2345"},
+ "type" : "ACKNOWLEDGE",
+ "acknowledge" : {
+ "agent_id" : {"value" : "12220-3440-12532-S1233"},
+ "task_id" : {"value" : "12220-3440-12532-my-task"},
+ "uuid" : "jhadf73jhakdlfha723adf"
}
}
@@ -297,12 +300,12 @@ Host: masterhost:5050
Content-Type: application/json
{
- âframework_idâ : {âvalueâ : â12220-3440-12532-2345â},
- âtypeâ : âRECONCILEâ,
- âreconcileâ : {
- âtasksâ : [
- { âtask_idâ : { âvalueâ : â312325â },
- âagent_idâ : { âvalueâ : â123535â }
+ "framework_id" : {"value" : "12220-3440-12532-2345"},
+ "type" : "RECONCILE",
+ "reconcile" : {
+ "tasks" : [
+ { "task_id" : { "value" : "312325" },
+ "agent_id" : { "value" : "123535" }
}
]
}
@@ -324,12 +327,12 @@ Host: masterhost:5050
Content-Type: application/json
{
- âframework_idâ : {âvalueâ : â12220-3440-12532-2345â},
- âtypeâ : âMESSAGEâ,
- âmessageâ : {
- âagent_idâ : {âvalueâ : â12220-3440-12532-S1233â},
- âexecutor_idâ : {âvalueâ : âmy-framework-executorâ},
- âdataâ : âadaf838jahd748jnaldfâ
+ "framework_id" : {"value" : "12220-3440-12532-2345"},
+ "type" : "MESSAGE",
+ "message" : {
+ "agent_id" : {"value" : "12220-3440-12532-S1233"},
+ "executor_id" : {"value" : "my-framework-executor"},
+ "data" : "adaf838jahd748jnaldf"
}
}
@@ -349,12 +352,12 @@ Host: masterhost:5050
Content-Type: application/json
{
- âframework_idâ : {âvalueâ : â12220-3440-12532-2345â},
- âtypeâ : âREQUESTâ,
- ârequestsâ : [
+ "framework_id" : {"value" : "12220-3440-12532-2345"},
+ "type" : "REQUEST",
+ "requests" : [
{
- âagent_idâ : {âvalueâ : â12220-3440-12532-S1233â},
- âresourcesâ : {}
+ "agent_id" : {"value" : "12220-3440-12532-S1233"},
+ "resources" : {}
},
]
}
@@ -366,7 +369,7 @@ HTTP/1.1 202 Accepted
## Events
-Scheduler is expected to keep a **persistent** connection open to â/schedulerâ endpoint even after getting a SUBSCRIBED HTTP Response event. This is indicated by âConnection: keep-aliveâ and âTransfer-Encoding: chunkedâ headers with *no* âContent-Lengthâ header set. All subsequent events that are relevant to this framework generated by Mesos are streamed on this connection. Master encodes each Event in RecordIO format, i.e., string representation of length of the event in bytes followed by JSON or binary Protobuf (possibly compressed) encoded event. Note that the value of length will never be â0â and the size of the length will be the size of unsigned integer (i.e., 64 bits). Also, note that the RecordIO encoding should be decoded by the scheduler whereas the underlying HTTP chunked encoding is typically invisible at the application (scheduler) layer. The type of content encoding used for the events will be determined by the accept
header of the POST request (e.g., Accept: application/json).
+Scheduler is expected to keep a **persistent** connection open to "/scheduler" endpoint even after getting a SUBSCRIBED HTTP Response event. This is indicated by "Connection: keep-alive" and "Transfer-Encoding: chunked" headers with *no* "Content-Length" header set. All subsequent events that are relevant to this framework generated by Mesos are streamed on this connection. Master encodes each Event in RecordIO format, i.e., string representation of length of the event in bytes followed by JSON or binary Protobuf (possibly compressed) encoded event. Note that the value of length will never be '0' and the size of the length will be the size of unsigned integer (i.e., 64 bits). Also, note that the RecordIO encoding should be decoded by the scheduler whereas the underlying HTTP chunked encoding is typically invisible at the application (scheduler) layer. The type of content encoding used for the events will be determined by the accept header of the POST request (e.g., Accept: applica
tion/json).
The following events are currently sent by the master. The canonical source of this information is at [scheduler.proto](include/mesos/v1/scheduler/scheduler.proto). Note that when sending JSON encoded events, master encodes raw bytes in Base64 and strings in UTF-8.
@@ -375,22 +378,22 @@ The first event sent by the master when
### OFFERS
-Sent by the master whenever there are new resources that can be offered to the framework. Each offer corresponds to a set of resources on a agent. Until the scheduler 'Accept's or 'Decline's an offer the resources are considered allocated to the scheduler, unless the offer is otherwise rescinded, e.g. due to a lost agent or `--offer_timeout`.
+Sent by the master whenever there are new resources that can be offered to the framework. Each offer corresponds to a set of resources on an agent. Until the scheduler 'Accept's or 'Decline's an offer the resources are considered allocated to the scheduler, unless the offer is otherwise rescinded, e.g. due to a lost agent or `--offer_timeout`.
```
OFFERS Event (JSON)
<event-length>
{
- âtypeâ : âOFFERSâ,
- âoffersâ : [
+ "type" : "OFFERS",
+ "offers" : [
{
- âoffer_idâ:{âvalueâ: â12214-23523-O235235â},
- âframework_idâ:{âvalueâ: â12124-235325-32425â},
- âagent_idâ:{âvalueâ: â12325-23523-S23523â},
- âhostnameâ:âagent.hostâ,
- âresourcesâ:[...],
- âattributesâ:[...],
- âexecutor_idsâ:[]
+ "offer_id":{"value": "12214-23523-O235235"},
+ "framework_id":{"value": "12124-235325-32425"},
+ "agent_id":{"value": "12325-23523-S23523"},
+ "hostname":"agent.host",
+ "resources":[...],
+ "attributes":[...],
+ "executor_ids":[]
}
]
}
@@ -403,9 +406,9 @@ Sent by the master when a particular off
RESCIND Event (JSON)
<event-length>
{
- âtypeâ : âRESCINDâ,
- ârescindâ : {
- âoffer_idâ : { âvalueâ : â12214-23523-O235235â}
+ "type" : "RESCIND",
+ "rescind" : {
+ "offer_id" : { "value" : "12214-23523-O235235"}
}
}
```
@@ -419,13 +422,13 @@ UPDATE Event (JSON)
<event-length>
{
- âtypeâ : âUPDATEâ,
- âupdateâ : {
- âstatusâ : {
- âtask_idâ : { âvalueâ : â12344-my-taskâ},
- âstateâ : âTASK_RUNNINGâ,
- âsourceâ : âSOURCE_EXECUTORâ,
- âuuidâ : âadfadfadbhgvjayd23r2uahjâ,
+ "type" : "UPDATE",
+ "update" : {
+ "status" : {
+ "task_id" : { "value" : "12344-my-task"},
+ "state" : "TASK_RUNNING",
+ "source" : "SOURCE_EXECUTOR",
+ "uuid" : "adfadfadbhgvjayd23r2uahj",
"bytes" : "uhdjfhuagdj63d7hadkf"
}
@@ -441,29 +444,29 @@ MESSAGE Event (JSON)
<event-length>
{
- âtypeâ : âMESSAGEâ,
- âmessageâ : {
- âagent_idâ : { âvalueâ : â12214-23523-S235235â},
- âexecutor_idâ : { âvalueâ : â12214-23523-my-executorâ},
- âdataâ : âadfadf3t2wa3353dfadfâ
+ "type" : "MESSAGE",
+ "message" : {
+ "agent_id" : { "value" : "12214-23523-S235235"},
+ "executor_id" : { "value" : "12214-23523-my-executor"},
+ "data" : "adfadf3t2wa3353dfadf"
}
}
```
### FAILURE
-Sent by the master when a agent is removed from the cluster (e.g., failed health checks) or when an executor is terminated. Note that, this event coincides with receipt of terminal `UPDATE` events for any active tasks belonging to the agent or executor and receipt of `RESCIND` events for any outstanding offers belonging to the agent. Note that there is no guaranteed order between the `FAILURE`, `UPDATE` and `RESCIND` events.
+Sent by the master when an agent is removed from the cluster (e.g., failed health checks) or when an executor is terminated. Note that, this event coincides with receipt of terminal `UPDATE` events for any active tasks belonging to the agent or executor and receipt of `RESCIND` events for any outstanding offers belonging to the agent. Note that there is no guaranteed order between the `FAILURE`, `UPDATE` and `RESCIND` events.
```
FAILURE Event (JSON)
<event-length>
{
- âtypeâ : âFAILUREâ,
- âfailureâ : {
- âagent_idâ : { âvalueâ : â12214-23523-S235235â},
- âexecutor_idâ : { âvalueâ : â12214-23523-my-executorâ},
- âstatusâ : 1
+ "type" : "FAILURE",
+ "failure" : {
+ "agent_id" : { "value" : "12214-23523-S235235"},
+ "executor_id" : { "value" : "12214-23523-my-executor"},
+ "status" : 1
}
}
```
@@ -476,8 +479,8 @@ ERROR Event (JSON)
<event-length>
{
- âtypeâ : âERRORâ,
- âmessageâ : âFramework is not authorizedâ
+ "type" : "ERROR",
+ "message" : "Framework is not authorized"
}
```
@@ -489,39 +492,39 @@ HEARTBEAT Event (JSON)
<event-length>
{
- âtypeâ : âHEARTBEATâ,
+ "type" : "HEARTBEAT",
}
```
## Disconnections
-Master considers a scheduler disconnected if the persistent subscription connection (opened via `SUBSCRIBE` request) to â/schedulerâ breaks. The connection could break for several reasons, e.g., scheduler restart, scheduler failover, network error. Note that the master doesnât keep track of non-subscription connection(s) to
-â/schedulerâ because it is not expected to be a persistent connection.
+Master considers a scheduler disconnected if the persistent subscription connection (opened via `SUBSCRIBE` request) to "/scheduler" breaks. The connection could break for several reasons, e.g., scheduler restart, scheduler failover, network error. Note that the master doesn't keep track of non-subscription connection(s) to
+"/scheduler" because it is not expected to be a persistent connection.
-If master realizes that the subscription connection is broken, it marks the scheduler as âdisconnectedâ and starts a failover timeout (failover timeout is part of FrameworkInfo). It also drops any pending events in its queue. Additionally, it rejects subsequent non-subscribe HTTP requests to â/schedulerâ with â403 Forbiddenâ, until the scheduler subscribes again with â/schedulerâ. If the scheduler *does not* re-subscribe within the failover timeout, the master considers the scheduler gone forever and shuts down all its executors, thus killing all its tasks. Therefore, all production schedulers are recommended to use a high value (e.g., 4 weeks) for the failover timeout.
+If master realizes that the subscription connection is broken, it marks the scheduler as "disconnected" and starts a failover timeout (failover timeout is part of FrameworkInfo). It also drops any pending events in its queue. Additionally, it rejects subsequent non-subscribe HTTP requests to "/scheduler" with "403 Forbidden", until the scheduler subscribes again with "/scheduler". If the scheduler *does not* re-subscribe within the failover timeout, the master considers the scheduler gone forever and shuts down all its executors, thus killing all its tasks. Therefore, all production schedulers are recommended to use a high value (e.g., 4 weeks) for the failover timeout.
-NOTE: To force shutdown a framework before the framework timeout elapses (e.g., during framework development and testing), either the framework can send `TEARDOWN` call (part of Scheduler API) or an operator can use the â/master/teardownâ endpoint (part of Operator API).
+NOTE: To force shutdown a framework before the framework timeout elapses (e.g., during framework development and testing), either the framework can send `TEARDOWN` call (part of Scheduler API) or an operator can use the "/master/teardown" endpoint (part of Operator API).
-If the scheduler realizes that its subscription connection to â/schedulerâ is broken, it should attempt to open a new persistent connection to the
-â/schedulerâ (on possibly new master based on the result of master detection) and resubscribe. It should not send new non-subscribe HTTP requests to â/schedulerâ unless it gets a `SUBSCRIBED` event; such requests will result in â403 Forbiddenâ.
+If the scheduler realizes that its subscription connection to "/scheduler" is broken, it should attempt to open a new persistent connection to the
+"/scheduler" (on possibly new master based on the result of master detection) and resubscribe. It should not send new non-subscribe HTTP requests to "/scheduler" unless it gets a `SUBSCRIBED` event; such requests will result in "403 Forbidden".
If the master does not realize that the subscription connection is broken, but the scheduler realizes it, the scheduler might open a new persistent connection to
-â/schedulerâ via `SUBSCRIBE`. In this case, the semantics depend on the value of `subscribe.force`. If set to true, master closes the existing subscription connection and allows subscription on the new connection. If set to false, the new connection attempt is disallowed in favor of the existing connection. The invariant here is that, only one persistent subscription connection for a given FrameworkID is allowed on the master. For HA schedulers, it is recommended that a scheduler instance set `subscribe.force` to true only when it just got elected and set it to false for all subsequent reconnection attempts (e.g, due to disconnection or master failover).
+"/scheduler" via `SUBSCRIBE`. In this case, the semantics depend on the value of `subscribe.force`. If set to true, master closes the existing subscription connection and allows subscription on the new connection. If set to false, the new connection attempt is disallowed in favor of the existing connection. The invariant here is that, only one persistent subscription connection for a given FrameworkID is allowed on the master. For HA schedulers, it is recommended that a scheduler instance set `subscribe.force` to true only when it just got elected and set it to false for all subsequent reconnection attempts (e.g, due to disconnection or master failover).
### Network partitions
-In the case of a network partition, the subscription connection between the scheduler and master might not necessarily break. To be able to detect this scenario, master periodically (e.g., 15s) sends `HEARTBEAT` events (similar in vein to Twitterâs Streaming API). If a scheduler doesnât receive a bunch (e.g., 5) of these heartbeats within a time window, it should immediately disconnect and try to re-subscribe. It is highly recommended for schedulers to use an exponential backoff strategy (e.g., upto a maximum of 15s) to avoid overwhelming the master while reconnecting. Schedulers can use a similar timeout (e.g., 75s) for receiving responses to any HTTP requests.
+In the case of a network partition, the subscription connection between the scheduler and master might not necessarily break. To be able to detect this scenario, master periodically (e.g., 15s) sends `HEARTBEAT` events (similar in vein to Twitter's Streaming API). If a scheduler doesn't receive a bunch (e.g., 5) of these heartbeats within a time window, it should immediately disconnect and try to re-subscribe. It is highly recommended for schedulers to use an exponential backoff strategy (e.g., upto a maximum of 15s) to avoid overwhelming the master while reconnecting. Schedulers can use a similar timeout (e.g., 75s) for receiving responses to any HTTP requests.
## Master detection
-Mesos has a high-availability mode that uses multiple Mesos masters; one active master (called the leader or leading master) and several standbys in case it fails. The masters elect the leader, with ZooKeeper coordinating the election. For more details please refer to the [documentation](/documentation/latest/high-availability.md).
+Mesos has a high-availability mode that uses multiple Mesos masters; one active master (/documentation/latest/called the leader or leading master) and several standbys in case it fails. The masters elect the leader, with ZooKeeper coordinating the election. For more details please refer to the [documentation](high-availability/).
-Schedulers are expected to make HTTP requests to the leading master. If requests are made to a non-leading master a âHTTP 307 Temporary Redirectâ will be received with the âLocationâ header pointing to the leading master.
+Schedulers are expected to make HTTP requests to the leading master. If requests are made to a non-leading master a "HTTP 307 Temporary Redirect" will be received with the "Location" header pointing to the leading master.
Example subscription workflow with redirection when the scheduler hits a non-leading master.
```
-Scheduler â Master
+Scheduler -> Master
POST /api/v1/scheduler HTTP/1.1
Host: masterhost1:5050
@@ -530,19 +533,19 @@ Accept: application/json
Connection: keep-alive
{
- âframework_infoâ : {
- âuserâ : âfooâ,
- ânameâ : âExample HTTP Frameworkâ
+ "framework_info" : {
+ "user" : "foo",
+ "name" : "Example HTTP Framework"
},
- âtypeâ : âSUBSCRIBEâ
+ "type" : "SUBSCRIBE"
}
-Master â Scheduler
+Master -> Scheduler
HTTP/1.1 307 Temporary Redirect
Location: masterhost2:5050
-Scheduler â Master
+Scheduler -> Master
POST /api/v1/scheduler HTTP/1.1
Host: masterhost2:5050
@@ -551,12 +554,12 @@ Accept: application/json
Connection: keep-alive
{
- âframework_infoâ : {
- âuserâ : âfooâ,
- ânameâ : âExample HTTP Frameworkâ
+ "framework_info" : {
+ "user" : "foo",
+ "name" : "Example HTTP Framework"
},
- âtypeâ : âSUBSCRIBEâ
+ "type" : "SUBSCRIBE"
}
```
-If the scheduler knows the list of masterâs hostnames for a cluster, it could use this mechanism to find the leading master to subscribe with. Alternatively, the scheduler could use a pure language library that detects the leading master given a ZooKeeper (or etcd) URL. For a `C++` library that does ZooKeeper based master detection please look at `src/scheduler/scheduler.cpp`.
+If the scheduler knows the list of master's hostnames for a cluster, it could use this mechanism to find the leading master to subscribe with. Alternatively, the scheduler could use a pure language library that detects the leading master given a ZooKeeper (or etcd) URL. For a `C++` library that does ZooKeeper based master detection please look at `src/scheduler/scheduler.cpp`.
Modified: mesos/site/source/documentation/latest/slave-recovery.md
URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/slave-recovery.md?rev=1727886&r1=1727885&r2=1727886&view=diff
==============================================================================
--- mesos/site/source/documentation/latest/slave-recovery.md (original)
+++ mesos/site/source/documentation/latest/slave-recovery.md Mon Feb 1 02:49:25 2016
@@ -13,7 +13,7 @@ Mesos slave could be restarted for an up
## How does it work?
-Slave recovery works by having the slave checkpoint enough information (e.g., Task Info, Executor Info, Status Updates) about the running tasks and executors to local disk. Once the slave ***and*** the framework(s) enable checkpointing, any subsequent slave restarts would recover the checkpointed information and reconnect with the executors. Note that if the host running the slave process is rebooted all the executors/tasks are killed.
+Slave recovery works by having the slave checkpoint enough information (e.g., Task Info, Executor Info, Status Updates) about the running tasks and executors to local disk. Once a framework enables checkpointing, any subsequent slave restarts would recover the checkpointed information and reconnect with the executors. Note that if the host running the slave process is rebooted all the executors/tasks are killed.
> NOTE: To enable recovery the framework should explicitly request checkpointing.
> Alternatively, a framework that doesn't want the disk i/o overhead of checkpointing can opt out of checkpointing.
@@ -60,9 +60,6 @@ Therefore, it is highly recommended to a
As part of this feature, `FrameworkInfo` has been updated to include an optional `checkpoint` field. A framework that would like to opt in to checkpointing should set `FrameworkInfo.checkpoint=True` before registering with the master.
-> NOTE: Frameworks that have enabled checkpointing will only get offers from checkpointing slaves. So, before setting `checkpoint=True` on FrameworkInfo, ensure that there are slaves in your cluster that have enabled checkpointing.
-> Because, if there are no checkpointing slaves, the framework would not get any offers and hence cannot launch any tasks/executors!
-
## Known issues with `systemd` and POSIX isolation
There is a known issue when using `systemd` to launch the `mesos-slave` while also using only `posix` isolation mechanisms that prevents tasks from recovering. The problem is that the default [KillMode](http://www.freedesktop.org/software/systemd/man/systemd.kill.html) for systemd processes is `cgroup` and hence all child processes are killed when the slave stops. Explicitly setting `KillMode` to `process` allows the executors to survive and reconnect.
@@ -80,4 +77,4 @@ KillMode=process
## Upgrading to 0.14.0
-If you want to upgrade a running Mesos cluster to 0.14.0 to take advantage of slave recovery please follow the [upgrade instructions](upgrades.md).
+If you want to upgrade a running Mesos cluster to 0.14.0 to take advantage of slave recovery please follow the [upgrade instructions](/documentation/latest/upgrades/).
Modified: mesos/site/source/documentation/latest/submitting-a-patch.md
URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/submitting-a-patch.md?rev=1727886&r1=1727885&r2=1727886&view=diff
==============================================================================
--- mesos/site/source/documentation/latest/submitting-a-patch.md (original)
+++ mesos/site/source/documentation/latest/submitting-a-patch.md Mon Feb 1 02:49:25 2016
@@ -26,14 +26,14 @@ layout: documentation
6. Find a **shepherd** to collaborate on your patch. A shepherd is a Mesos committer that will work with you to give you feedback on your proposed design, and to eventually commit your change into the Mesos source tree.
1. To find a shepherd, email the dev mailing list (include a link to your JIRA issue). You can also try asking by adding a comment to your JIRA issue.
- 2. You can also find a shepherd by asking the developers on IRC (in the [mesos channel](irc://irc.freenode.net/mesos) on [Freenode](https://freenode.net)). You can find the current list of committers [here](committers.md): a developer that has previously worked on the component you are modifying might be a good candidate shepherd.
+ 2. You can also find a shepherd by asking the developers on IRC (/documentation/latest/in the [mesos channel](irc://irc.freenode.net/mesos) on [Freenode](https://freenode.net)). You can find the current list of committers [here](committers/): a developer that has previously worked on the component you are modifying might be a good candidate shepherd.
### Create your patch
1. Create one or more test cases to exercise the bug or the feature (the Mesos team uses [test-driven development](http://en.wikipedia.org/wiki/Test-driven_development)). Before you start coding, make sure these test cases all fail.
- 1. The [testing patterns](testing-patterns.md) page has some suggestions for writing test cases.
+ 1. The [testing patterns](/documentation/latest/testing-patterns/) page has some suggestions for writing test cases.
2. Make your changes to the code (using whatever IDE/editor you choose) to actually fix the bug or implement the feature.
- 1. Before beginning, please read the [Mesos C++ Style Guide](c++-style-guide.md). It is recommended to use the git pre-commit hook (`support/hooks/pre-commit`) to automatically check for style errors. See the hook script for instructions to enable it.
+ 1. Before beginning, please read the [Mesos C++ Style Guide](/documentation/latest/c++-style-guide/). It is recommended to use the git pre-commit hook (`support/hooks/pre-commit`) to automatically check for style errors. See the hook script for instructions to enable it.
2. Most of your changes will probably be to files inside of `BASE_MESOS_DIR`
3. From inside of the root Mesos directory: `./bootstrap` (Only required if building from git repository).
4. To build, we recommend that you don't build inside of the src directory. We recommend you do the following:
@@ -46,9 +46,10 @@ layout: documentation
1. To build all tests without executing them, use something like: `make tests`.
2. To execute a single unit test (helpful when trying to debug a test case failure), use something like: `make check GTEST_FILTER="HTTPTest.Delete"`.
-4. Divide your change into one or more Git commits. Each commit should represent a single logical (atomic) change to the Mesos source code: this makes your changes easier to review. For more information, see the [reviewer guidelines](effective-code-reviewing.md).
+4. Divide your change into one or more Git commits. Each commit should represent a single logical (/documentation/latest/atomic) change to the Mesos source code: this makes your changes easier to review. For more information, see the [reviewer guidelines](effective-code-reviewing/).
1. Try to avoid including other, unrelated cleanups (e.g., typo fixes or style nits) in the same commit that makes functional changes. While typo fixes are great, including them in the same commit as functional changes makes the commit history harder to read.
2. Developers often make incremental commits to save their progress when working on a change, and then "rewrite history" (e.g., using `git rebase -i`) to create a clean set of commits once the change is ready to be reviewed.
+ 3. Commit messages should be in past tense. The first sentence should summarize the change; it should start with a capital letter, not exceed 72 characters and end in a period.
5. Make sure to pull in any changes that have been committed to master branch. Using Git, do this via something like:
1. `git checkout master`
@@ -82,4 +83,4 @@ layout: documentation
4. The last step is to ensure that the necessary documentation gets created or updated so the whole world knows about your new feature or bug fix.
## Style Guides
-* For patches to the core, we ask that you follow the [Mesos C++ Style Guide](c++-style-guide.md).
+* For patches to the core, we ask that you follow the [Mesos C++ Style Guide](/documentation/latest/c++-style-guide/).
Modified: mesos/site/source/documentation/latest/testing-patterns.md
URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/testing-patterns.md?rev=1727886&r1=1727885&r2=1727886&view=diff
==============================================================================
--- mesos/site/source/documentation/latest/testing-patterns.md (original)
+++ mesos/site/source/documentation/latest/testing-patterns.md Mon Feb 1 02:49:25 2016
@@ -9,7 +9,7 @@ A collection of common testing patterns
## Expediting events with `Clock`
Some events in Mesos are separated by certain timeouts, for example framework registration attempts. Simple waiting for such events to fire leads to blocking the test thread for the duration of the associated timeout. This increases the duration of `make check` for no good reason.
-If an event is triggered by an act of processing a message from an actor's mailbox, it can be expedited with the help of libprocess' `Clock` routines. Delayed messages are maintained in sorted order by their due time and are dispatched â i.e. pushed into destination mailboxes â when this time comes. An important bit here is that time is driven by the internal libprocess clock. We can shift this clock into the future by calling `Clock::advance(<duration>)`, rendering certain front messages in the collection due now. These messages are dispatched instantly, effectively overriding the associated event's timeout.
+If an event is triggered by an act of processing a message from an actor's mailbox, it can be expedited with the help of libprocess' `Clock` routines. Delayed messages are maintained in sorted order by their due time and are dispatched - i.e. pushed into destination mailboxes - when this time comes. An important bit here is that time is driven by the internal libprocess clock. We can shift this clock into the future by calling `Clock::advance(<duration>)`, rendering certain front messages in the collection due now. These messages are dispatched instantly, effectively overriding the associated event's timeout.
**NOTE**: Without calling `Clock::settle()` there is no guarantee a dispatched message has been already processed.
Modified: mesos/site/source/documentation/latest/tools.md
URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/tools.md?rev=1727886&r1=1727885&r2=1727886&view=diff
==============================================================================
--- mesos/site/source/documentation/latest/tools.md (original)
+++ mesos/site/source/documentation/latest/tools.md Mon Feb 1 02:49:25 2016
@@ -9,7 +9,7 @@ layout: documentation
These tools make it easy to set up and run a Mesos cluster.
* [collectd plugin](https://github.com/rayrod2030/collectd-mesos) to collect Mesos cluster metrics.
-* [Deploy scripts](deploy-scripts.md) for launching a Mesos cluster on a set of machines.
+* [Deploy scripts](/documentation/latest/deploy-scripts/) for launching a Mesos cluster on a set of machines.
* [Chef cookbook by Everpeace](https://github.com/everpeace/cookbook-mesos) Install Mesos and configure master and slave. This cookbook supports installation from source or the Mesosphere packages.
* [Chef cookbook by Mdsol](https://github.com/mdsol/mesos_cookbook) Application cookbook for installing the Apache Mesos cluster manager. This cookbook installs Mesos via packages provided by Mesosphere.
* [Puppet Module by Deric](https://github.com/deric/puppet-mesos) This is a Puppet module for managing Mesos nodes in a cluster.
@@ -20,7 +20,7 @@ These tools make it easy to set up and r
If you want to hack on Mesos or write a new framework, these tools will help.
-* [clang-format](/documentation/latest/clang-format/) to automatically apply some of the style rules dictated by the [Mesos C++ Style Guide](/documentation/latest/c++-style-guide/).
+* [clang-format](/documentation/latest/clang-format.md) to automatically apply some of the style rules dictated by the [Mesos C++ Style Guide](c++-style-guide/).
* [Go Bindings and Examples](https://github.com/mesosphere/mesos-go) Write a Mesos framework in Go! Comes with an example scheduler and executor.
* [Mesos Framework giter8 Template](https://github.com/mesosphere/scala-sbt-mesos-framework.g8) This is a giter8 template. The result of applying this template is a bare-bones Apache Mesos framework in Scala using SBT for builds and Vagrant for testing on a singleton cluster.
* [Scala Hello World](https://gist.github.com/guenter/7471695) A simple Mesos "Hello World": downloads and starts a web server on every node in the cluster.
Modified: mesos/site/source/documentation/latest/upgrades.md
URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/upgrades.md?rev=1727886&r1=1727885&r2=1727886&view=diff
==============================================================================
--- mesos/site/source/documentation/latest/upgrades.md (original)
+++ mesos/site/source/documentation/latest/upgrades.md Mon Feb 1 02:49:25 2016
@@ -4,7 +4,21 @@ layout: documentation
# Upgrading Mesos
-This document serves as a guide for users who wish to upgrade an existing mesos cluster. Some versions require particular upgrade techniques when upgrading a running cluster. Some upgrades will have incompatible changes.
+This document serves as a guide for users who wish to upgrade an existing Mesos cluster. Some versions require particular upgrade techniques when upgrading a running cluster. Some upgrades will have incompatible changes.
+
+## Upgrading from 0.26.x to 0.27.x ##
+
+* Mesos 0.27 introduces the concept of _implicit roles_. In previous releases, configuring roles required specifying a static whitelist of valid role names on master startup (/documentation/latest/via the `--roles` flag). In Mesos 0.27, if `--roles` is omitted, _any_ role name can be used; controlling which principals are allowed to register as which roles should be done using [ACLs](authorization/). The role whitelist functionality is still supported but is deprecated.
+
+* The Allocator API has changed due to the introduction of implicit roles. Custom allocator implementations will need to be updated. See [MESOS-4000](https://issues.apache.org/jira/browse/MESOS-4000) for more information.
+
+* The `executorLost` callback in the Scheduler interface will now be called whenever the slave detects termination of a custom executor. This callback was never called in previous versions, so please make sure any framework schedulers can now safely handle this callback. Note that this callback may not be reliably delivered.
+
+* The isolator `prepare` interface has been changed slightly. Instead of keeping adding parameters to the `prepare` interface, we decide to use a protobuf (`ContainerConfig`). Also, we renamed `ContainerPrepareInfo` to `ContainerLaunchInfo` to better capture the purpose of this struct. See [MESOS-4240](https://issues.apache.org/jira/browse/MESOS-4240) and [MESOS-4282](https://issues.apache.org/jira/browse/MESOS-4282) for more information. If you are an isolator module writer, you will have to adjust your isolator module according to the new interface and re-compile with 0.27.
+
+* ACLs.shutdown_frameworks has been deprecated in favor of the new ACLs.teardown_frameworks. This affects the `--acls` master flag for the local authorizer.
+
+* Reserved resources are now accounted for in the DRF role sorter. Previously unaccounted reservations will influence the weighted DRF sorter. If role weights were explicitly set, they may need to be adjusted in order to account for the reserved resources in the cluster.
## Upgrading from 0.25.x to 0.26.x ##
@@ -21,7 +35,6 @@ On slaves, the affected `data` field was
**NOTE** The `NetworkInfo` protobuf has been changed. The fields `protocol` and `ip_address` are now deprecated. The new field `ip_addresses` subsumes the information provided by them.
-
## Upgrading from 0.24.x to 0.25.x
**NOTE** The following endpoints will be deprecated in favor of new endpoints. Both versions will be available in 0.25 but the deprecated endpoints will be removed in a subsequent release.
@@ -114,7 +127,7 @@ Please refer to the metrics/snapshot end
}
-**NOTE** All Mesos arguments can now be passed using file:// to read them out of a file (either an absolute or relative path). The --credentials, --whitelist, and any flags that expect JSON backed arguments (such as --modules) behave as before, although support for just passing a absolute path for any JSON flags rather than file:// has been deprecated and will produce a warning (and the absolute path behavior will be removed in a future release).
+**NOTE** All Mesos arguments can now be passed using file:// to read them out of a file (either an absolute or relative path). The --credentials, --whitelist, and any flags that expect JSON backed arguments (such as --modules) behave as before, although support for just passing an absolute path for any JSON flags rather than file:// has been deprecated and will produce a warning (and the absolute path behavior will be removed in a future release).
In order to upgrade a running cluster:
Added: mesos/site/source/documentation/latest/versioning.md
URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/versioning.md?rev=1727886&view=auto
==============================================================================
--- mesos/site/source/documentation/latest/versioning.md (added)
+++ mesos/site/source/documentation/latest/versioning.md Mon Feb 1 02:49:25 2016
@@ -0,0 +1,100 @@
+---
+layout: documentation
+---
+
+# Mesos Versioning
+
+The Mesos API and release versioning policy gives operators and developers clear guidelines on:
+
+* Making modifications to the existing APIs without affecting backward compatibility.
+* How long a Mesos API will be supported.
+* Upgrading the Mesos installation across release versions.
+
+API versioning was introduced in Mesos 0.24.0 and this scheme only applies to Mesos 1.0.0 and higher.
+
+## Terminology
+
+* **Release Versioning**: This refers to the version of Mesos that is being released. It is of the form **Mesos X.Y.Z** (X is the major version, Y is the minor version, and Z is the patch version).
+* **API Versioning**: This refers to the version of the Mesos API. It is of the form **vX** (X is the major version).
+
+## How does it work?
+
+The Mesos APIs (constituting Scheduler, Executor, Internal, Operator/Admin APIs) will have a version in the URL. The versioned URL will have a prefix of **`/api/vN`** where "N" is the version of the API. The "/api" prefix is chosen to distinguish API resources from Web UI paths.
+
+Examples:
+
+* http://localhost:5050/api/v1/scheduler : Scheduler HTTP API hosted by the master.
+* http://localhost:5051/api/v1/executor : Executor HTTP API hosted by the agent.
+
+A given Mesos installation might host multiple versions of the same API i.e., Scheduler API v1 and/or v2 etc.
+
+### API version vs Release version
+
+* To keep things simple, the stable version of the API will correspond to the major release version of Mesos.
+ * For example, v1 of the API will be supported by Mesos release versions 1.0.0, 1.4.0, 1.20.0 etc.
+* vN version of the API might also be supported by release versions of N-1 series but the vN API is not considered stable until the last release version of N-1 series.
+ * For example, v2 of the API might be introduced in Mesos 1.12.0 release but it is only considered stable in Mesos 1.21.0 release if it is the last release of â1â series. Note that all Mesos 1.x.y versions will still support v1 of the API.
+* The API version is only bumped if we need to make a backwards [incompatible](#api-compatibility) API change. We will strive to support a given API version for at least a year.
+* The deprecation clock for vN-1 API will start as soon as we release âN.0.0â version of Mesos. We will strive to give enough time (e.g., 6 months) for frameworks/operators to upgrade to vN API before we stop supporting vN-1 API.
+* Minor release version is bumped roughly on a monthly cycle to give a cadence of new features to users.
+
+NOTE: Presently, for "0.X.Y" releases i.e. till we reach "1.0.0", we wait for at least 6 monthly releases before deprecating functionality. So, functionality that has been deprecated in 0.26.0 can be safely removed in 0.32.0.
+
+### API Compatibility
+
+The API compatibility is determined by the corresponding protobuf guarantees.
+
+As an example, the following are considered "backwards compatible" changes for Scheduler API:
+
+* Adding new types of Calls i.e., new types of HTTP requests to "/scheduler".
+* Adding new optional fields to existing requests to "/scheduler".
+* Adding new types of Events i.e., new types of chunks streamed on "/scheduler".
+* Adding new header fields to chunked response streamed on "/scheduler".
+* Adding new fields (or changing the order of fields) to chunksâ body streamed on "/scheduler".
+* Adding new API resources (e.g., "/foobar").
+
+The following are considered backwards incompatible changes for Scheduler API:
+
+* Adding new required fields to existing requests to "/scheduler".
+* Renaming/removing fields from existing requests to "/scheduler".
+* Renaming/removing fields from chunks streamed on "/scheduler".
+* Renaming/removing existing Calls.
+
+## Upgrades
+
+* The master and agent are typically compatible as long as they are running the same major version.
+ * For example, 1.3.0 master is compatible with 1.13.0 agent.
+* In rare cases, we might require the master and agent to go through a specific minor version for upgrades.
+ * For example, we might require that a 1.1.0 master (and/or agent) be upgraded to 1.8.0 before it can be upgraded to 1.9.0 or later versions.
+
+The detailed information about upgrading to a particular Mesos version would be posted [here](/documentation/latest/upgrades/).
+
+### Implementation Details
+
+Most APIs in Mesos accept protobuf messages with a corresponding JSON field mapping. To support multiple versions of the API, we decoupled the versioned protobufs backing the API from the âinternalâ protobufs used by the Mesos code.
+
+For example, the protobufs for the v1 Scheduler API are located at:
+
+```
+include/mesos/v1/scheduler/scheduler.proto
+
+package mesos.v1.scheduler;
+option java_package = "org.apache.mesos.v1.scheduler";
+option java_outer_classname = "Protos";
+â¦
+```
+
+The corresponding internal protobufs for the Scheduler API are located at:
+
+```
+include/mesos/scheduler/scheduler.proto
+
+package mesos.scheduler;
+option java_package = "org.apache.mesos.scheduler";
+option java_outer_classname = "Protos";
+â¦
+```
+
+The users of the API send requests (and receive responses) based on the versioned protobufs. We implemented [evolve](https://github.com/apache/mesos/blob/master/src/internal/evolve.hpp)/[devolve](https://github.com/apache/mesos/blob/master/src/internal/devolve.hpp) converters that can convert protobufs from any supported version to the internal protobuf and vice versa.
+
+Internally, message passing between various Mesos components would use the internal unversioned protobufs. When sending response (if any) back to the user of the API, the unversioned protobuf would be converted back to a versioned protobuf.
Modified: mesos/site/source/downloads.html.md
URL: http://svn.apache.org/viewvc/mesos/site/source/downloads.html.md?rev=1727886&r1=1727885&r2=1727886&view=diff
==============================================================================
--- mesos/site/source/downloads.html.md (original)
+++ mesos/site/source/downloads.html.md Mon Feb 1 02:49:25 2016
@@ -8,6 +8,10 @@ breadcrumb: Downloads
<h4>Previous Releases</h4>
<ul>
+ <li><a href="http://archive.apache.org/dist/mesos/0.26.0/">0.26.0</a>
+ (<a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311242&version=12333528">Release Notes</a>)</li>
+ <li><a href="http://archive.apache.org/dist/mesos/0.25.0/">0.25.0</a>
+ (<a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311242&version=12329852">Release Notes</a>)</li>
<li><a href="http://archive.apache.org/dist/mesos/0.24.1/">0.24.1</a>
(<a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311242&version=12333552">Release Notes</a>)</li>
<li><a href="http://archive.apache.org/dist/mesos/0.24.0/">0.24.0</a>
@@ -70,8 +74,8 @@ breadcrumb: Downloads
<div class="col-md-8">
<h1>Download Mesos</h1>
<p>Download the most recent stable release:
- <a href="http://www.apache.org/dyn/mirrors/mirrors.cgi/mesos/0.25.0/">0.25.0</a>
- (<a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311242&version=12329852">Release Notes</a>)
+ <a href="http://www.apache.org/dyn/mirrors/mirrors.cgi/mesos/0.27.0/">0.27.0</a>
+ (<a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311242&version=12334043">Release Notes</a>)
</p>
<h4>Getting the code via source control</h4>
Modified: mesos/site/source/index.html.md
URL: http://svn.apache.org/viewvc/mesos/site/source/index.html.md?rev=1727886&r1=1727885&r2=1727886&view=diff
==============================================================================
--- mesos/site/source/index.html.md (original)
+++ mesos/site/source/index.html.md Mon Feb 1 02:49:25 2016
@@ -4,21 +4,10 @@ title: Apache Mesos
<div class="container-fluid homepage jumbotron">
<div class="row">
- <div class="col-md-7">
- <h1>Program against your datacenter like itâs a single pool of resources</h1>
- <p class="lead">Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively.</p>
+ <h1>Program against your datacenter like itâs a single pool of resources</h1>
+ <p class="lead">Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively.</p>
- <p> <a href="downloads/"><span class="glyphicon glyphicon-download"></span> Download Mesos 0.25.0</a> or learn how to <a href="gettingstarted/">get started</a></p>
- </div>
- <div class="col-md-5 text-center download">
- <div class="homepage-featured">
- <h2>MesosCon Europe</h2>
- <p>Dublin, October 8 - 9th, 2015</p>
- #MesosCon Europe is conference organized by the Apache Mesos community, bringing together users and developers to share and learn about the project and its growing ecosystem.
- <br /><br />
- <a class="btn btn-default btn-success" href="http://events.linuxfoundation.org/events/mesoscon-europe/attend/register" role="button">REGISTER TODAY</a>
- </div>
- </div>
+ <p> <a href="downloads/"><span class="glyphicon glyphicon-download"></span> Download Mesos 0.27.0</a> or learn how to <a href="gettingstarted/">get started</a></p>
</div>
</div>
@@ -43,6 +32,8 @@ title: Apache Mesos
<div class="col-md-5">
<h3>News</h3>
<ul>
+ <li><em>January 31, 2016</em> - Mesos 0.27.0 is released! See the <a href="https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.27.0">CHANGELOG</a> for details.</li>
+ <li><em>December 16, 2015</em> - Mesos 0.26.0 is released! See the <a href="https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.26.0">CHANGELOG</a> and <a href="/blog/mesos-0-26-0-released/">blog post</a> for details.</li>
<li><em>October 10, 2015</em> - Mesos 0.25.0 is released! See the <a href="https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.25.0">CHANGELOG</a> and <a href="/blog/mesos-0-25-0-released/">blog post</a> for details.</li>
<li><em>September 30, 2015</em> - Announcing the #MesosCon Europe Hackathon! See the <a href="/blog/announcing-mesoscon-eu-2015-hackathon/">blog post</a> for details.</li>
<li><em>September 24, 2015</em> - Mesos 0.24.1 is released! See the <a href="https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.24.1">CHANGELOG</a> and <a href="/blog/mesos-0-24-1-and-more-released/">blog post</a> for details.</li>