You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mesos.apache.org by dl...@apache.org on 2014/08/25 01:10:34 UTC
svn commit: r1620233 [2/4] - in /mesos/site: publish/
publish/api/latest/java/ publish/api/latest/java/org/apache/mesos/
publish/documentation/ publish/documentation/allocation-module/
publish/documentation/authorization/ publish/documentation/docker-c...
Added: mesos/site/publish/documentation/latest/docker-containerizer/index.html
URL: http://svn.apache.org/viewvc/mesos/site/publish/documentation/latest/docker-containerizer/index.html?rev=1620233&view=auto
==============================================================================
--- mesos/site/publish/documentation/latest/docker-containerizer/index.html (added)
+++ mesos/site/publish/documentation/latest/docker-containerizer/index.html Sun Aug 24 23:10:33 2014
@@ -0,0 +1,186 @@
+<!DOCTYPE html>
+<!--
+
+ ______ __
+ /\ _ \ /\ \
+ \ \ \L\ \ _____ __ ___\ \ \___ __
+ \ \ __ \/\ '__`\ /'__`\ /'___\ \ _ `\ /'__`\
+ \ \ \/\ \ \ \L\ \/\ \L\.\_/\ \__/\ \ \ \ \/\ __/
+ \ \_\ \_\ \ ,__/\ \__/.\_\ \____\\ \_\ \_\ \____\
+ \/_/\/_/\ \ \/ \/__/\/_/\/____/ \/_/\/_/\/____/
+ \ \_\
+ \/_/
+
+ /'\_/`\
+ /\ \ __ ____ ___ ____
+ \ \ \__\ \ /'__`\ /',__\ / __`\ /',__\
+ \ \ \_/\ \/\ __//\__, `\/\ \L\ \/\__, `\
+ \ \_\\ \_\ \____\/\____/\ \____/\/\____/
+ \/_/ \/_/\/____/\/___/ \/___/ \/___/
+
+-->
+<html>
+ <head>
+ <meta charset="utf-8">
+ <title></title>
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+ <meta name="description" content="">
+ <meta name="author" content="">
+
+ <!-- Le styles -->
+ <link href="//netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
+
+ <link href="../../../assets/css/main.css" media="screen" rel="stylesheet" type="text/css" />
+
+
+
+ <!-- Google Analytics Magic -->
+ <script type="text/javascript">
+ var _gaq = _gaq || [];
+ _gaq.push(['_setAccount', 'UA-20226872-1']);
+ _gaq.push(['_setDomainName', 'apache.org']);
+ _gaq.push(['_trackPageview']);
+
+ (function() {
+ var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
+ ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
+ var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
+ })();
+ </script>
+ </head>
+ <body>
+ <!-- magical breadcrumbs -->
+ <div class="topnav">
+ <ul class="breadcrumb">
+ <li>
+ <div class="dropdown">
+ <a data-toggle="dropdown" href="#">Apache Software Foundation <span class="caret"></span></a>
+ <ul class="dropdown-menu" role="menu" aria-labelledby="dLabel">
+ <li><a href="http://www.apache.org">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ </ul>
+ </div>
+ </li>
+ <li><a href="http://mesos.apache.org">Apache Mesos</a></li>
+
+
+ <li><a href="/documentation
+/">Documentation
+</a></li>
+
+
+ </ul><!-- /breadcrumb -->
+ </div>
+
+ <!-- navbar excitement -->
+ <div class="navbar navbar-static-top" role="navigation">
+ <div class="navbar-inner">
+ <div class="container">
+ <a href="/" class="logo"><img src="/assets/img/mesos_logo.png" alt="Apache Mesos logo" /></a>
+ <div class="nav-collapse">
+ <ul class="nav nav-pills navbar-right">
+ <li><a href="/gettingstarted/">Getting Started</a></li>
+ <li><a href="/documentation/latest/">Documentation</a></li>
+ <li><a href="/downloads/">Downloads</a></li>
+ <li><a href="/community/">Community</a></li>
+ </ul>
+ </div>
+ </div>
+ </div>
+ </div><!-- /.navbar -->
+
+ <div class="container">
+
+ <div class="row-fluid">
+ <div class="col-md-4">
+ <h4>If you're new to Mesos</h4>
+ <p>See the <a href="/gettingstarted/">getting started</a> page for more information about downloading, building, and deploying Mesos.</p>
+
+ <h4>If you'd like to get involved or you're looking for support</h4>
+ <p>See our <a href="/community/">community</a> page for more details.</p>
+ </div>
+ <div class="col-md-8">
+ <h1>Docker Containerizer</h1>
+
+<p>Mesos 0.20.0 adds the support for launching tasks that contains Docker images, with also a subset of Docker options supported while we plan on adding more in the future.</p>
+
+<p>Users can either launch a Docker image as a Task, or as an Executor.</p>
+
+<p>The following sections will describe the API changes along with Docker support, and also how to setup Docker.</p>
+
+<h2>Setup</h2>
+
+<p>To run the slave to enable the Docker Containerizer, you must launch the slave with “docker” as one of the containerizers option.</p>
+
+<p>Example: mesos-slave –containerizers=docker,mesos</p>
+
+<p>Each slave that has the Docker containerizer should have Docker CLI client installed (version >= 1.0.0).</p>
+
+<h2>How do I use the Docker Containerizer?</h2>
+
+<p>TaskInfo before 0.20.0 used to only support either setting a CommandInfo that launches a task running the bash command, or a ExecutorInfo that launches a custom Executor
+that will launches the task.</p>
+
+<p>With 0.20.0 we added a ContainerInfo field to TaskInfo and ExecutorInfo that allows a Containerizer such as Docker to be configured to run the task or executor.</p>
+
+<p>To run a Docker image as a task, in TaskInfo one must set both the command and the container field as the Docker Containerizer will use the accompanied command to launch the docker image.
+The ContainerInfo should have type Docker and a DockerInfo that has the desired docker image.</p>
+
+<p>To run a Docker image as an executor, in TaskInfo one must set the ExecutorInfo that contains a ContainerInfo with type docker and the CommandInfo that will be used to launch the executor.
+Note that the Docker image is expected to launch up as a Mesos executor that will register with the slave once it launches.</p>
+
+<h2>What does the Docker Containerizer do?</h2>
+
+<p>The Docker Containerizer is translating Task/Executor Launch and Destroy calls to Docker CLI commands.</p>
+
+<p>Currently the Docker Containerizer when launching as task will do the following:</p>
+
+<p>1, Fetch all the files specified in the CommandInfo into the sandbox.
+2, Pull the docker image from the remote repository.
+3, Run the docker image with the configured DockerInfo options, and map the sandbox directory into the Docker container and set the directory mapping to the MESOS_SANDBOX environment variable.
+4. Stream the docker logs into the stdout/stderr files in the sandbox.
+5. Launch the Command Executor to perform a docker wait on the container.
+6. On container exit or containerizer destroy, stop and remove the docker container.</p>
+
+<p>When launching the docker image as an Executor, the only difference is that it skips launching a command executor but just reaps on the docker container executor pid.</p>
+
+<p>Note that we currently default to host networking when running a docker image, to easier support running a docker image asn an Executor.</p>
+
+<p>Also since we explicitly attempt to pull the image on launch, if the docker image is only installed locally but not avaialble on the remote repository the launch will fail as well.</p>
+
+<h2>Private Docker repository</h2>
+
+<p>To run a image from a private repository, one can include the uri pointing to a .dockercfg that contains login information. The .dockercfg file will be pulled into the sandbox the Docker Containerizer
+set the HOME environment variable pointing to the sandbox so docker cli will automatically pick up the config file.</p>
+
+<h2>CommandInfo to run Docker images</h2>
+
+<p>A docker image currently supports having a entrypoint and/or a default command.</p>
+
+<p>To run a docker image with the default command (ie: docker run image), the CommandInfo’s value must not be set. If the value is set then it will override the default command.</p>
+
+<p>To run a docker image with an entrypoint defined, the CommandInfo’s shell option must be set to false.
+If shell option is set to true the Docker Containerizer will run the user’s command wrapped with /bin/sh -c which will also become parameters to the image entrypoint.</p>
+
+ </div>
+</div>
+
+
+ <hr>
+
+ <!-- footer -->
+ <div class="footer">
+ <p>© 2012-2014 <a href="http://apache.org">The Apache Software Foundation</a>.
+ Apache Mesos, the Apache feather logo, and the Apache Mesos project logo are trademarks of The Apache Software Foundation.<p>
+ </div><!-- /footer -->
+
+ </div> <!-- /container -->
+
+ <!-- JS -->
+ <script src="//code.jquery.com/jquery-1.11.0.min.js" type="text/javascript"></script>
+ <script src="//netdna.bootstrapcdn.com/bootstrap/3.1.1/js/bootstrap.min.js" type="text/javascript"></script>
+ </body>
+</html>
\ No newline at end of file
Added: mesos/site/publish/documentation/latest/framework-rate-limiting/index.html
URL: http://svn.apache.org/viewvc/mesos/site/publish/documentation/latest/framework-rate-limiting/index.html?rev=1620233&view=auto
==============================================================================
--- mesos/site/publish/documentation/latest/framework-rate-limiting/index.html (added)
+++ mesos/site/publish/documentation/latest/framework-rate-limiting/index.html Sun Aug 24 23:10:33 2014
@@ -0,0 +1,221 @@
+<!DOCTYPE html>
+<!--
+
+ ______ __
+ /\ _ \ /\ \
+ \ \ \L\ \ _____ __ ___\ \ \___ __
+ \ \ __ \/\ '__`\ /'__`\ /'___\ \ _ `\ /'__`\
+ \ \ \/\ \ \ \L\ \/\ \L\.\_/\ \__/\ \ \ \ \/\ __/
+ \ \_\ \_\ \ ,__/\ \__/.\_\ \____\\ \_\ \_\ \____\
+ \/_/\/_/\ \ \/ \/__/\/_/\/____/ \/_/\/_/\/____/
+ \ \_\
+ \/_/
+
+ /'\_/`\
+ /\ \ __ ____ ___ ____
+ \ \ \__\ \ /'__`\ /',__\ / __`\ /',__\
+ \ \ \_/\ \/\ __//\__, `\/\ \L\ \/\__, `\
+ \ \_\\ \_\ \____\/\____/\ \____/\/\____/
+ \/_/ \/_/\/____/\/___/ \/___/ \/___/
+
+-->
+<html>
+ <head>
+ <meta charset="utf-8">
+ <title></title>
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+ <meta name="description" content="">
+ <meta name="author" content="">
+
+ <!-- Le styles -->
+ <link href="//netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
+
+ <link href="../../../assets/css/main.css" media="screen" rel="stylesheet" type="text/css" />
+
+
+
+ <!-- Google Analytics Magic -->
+ <script type="text/javascript">
+ var _gaq = _gaq || [];
+ _gaq.push(['_setAccount', 'UA-20226872-1']);
+ _gaq.push(['_setDomainName', 'apache.org']);
+ _gaq.push(['_trackPageview']);
+
+ (function() {
+ var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
+ ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
+ var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
+ })();
+ </script>
+ </head>
+ <body>
+ <!-- magical breadcrumbs -->
+ <div class="topnav">
+ <ul class="breadcrumb">
+ <li>
+ <div class="dropdown">
+ <a data-toggle="dropdown" href="#">Apache Software Foundation <span class="caret"></span></a>
+ <ul class="dropdown-menu" role="menu" aria-labelledby="dLabel">
+ <li><a href="http://www.apache.org">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ </ul>
+ </div>
+ </li>
+ <li><a href="http://mesos.apache.org">Apache Mesos</a></li>
+
+
+ <li><a href="/documentation
+/">Documentation
+</a></li>
+
+
+ </ul><!-- /breadcrumb -->
+ </div>
+
+ <!-- navbar excitement -->
+ <div class="navbar navbar-static-top" role="navigation">
+ <div class="navbar-inner">
+ <div class="container">
+ <a href="/" class="logo"><img src="/assets/img/mesos_logo.png" alt="Apache Mesos logo" /></a>
+ <div class="nav-collapse">
+ <ul class="nav nav-pills navbar-right">
+ <li><a href="/gettingstarted/">Getting Started</a></li>
+ <li><a href="/documentation/latest/">Documentation</a></li>
+ <li><a href="/downloads/">Downloads</a></li>
+ <li><a href="/community/">Community</a></li>
+ </ul>
+ </div>
+ </div>
+ </div>
+ </div><!-- /.navbar -->
+
+ <div class="container">
+
+ <div class="row-fluid">
+ <div class="col-md-4">
+ <h4>If you're new to Mesos</h4>
+ <p>See the <a href="/gettingstarted/">getting started</a> page for more information about downloading, building, and deploying Mesos.</p>
+
+ <h4>If you'd like to get involved or you're looking for support</h4>
+ <p>See our <a href="/community/">community</a> page for more details.</p>
+ </div>
+ <div class="col-md-8">
+ <h1>Framework Rate Limiting</h1>
+
+<p>Framework rate limiting is a feature introduced in Mesos 0.20.0.</p>
+
+<h2>What is Framework Rate Limiting</h2>
+
+<p>In a multi-framework environment, this feature aims to protect the throughput of high-SLA (e.g., production, service) frameworks by having the master throttle messages from other (e.g., development, batch) frameworks.</p>
+
+<p>To throttle messages from a framework, the Mesos cluster operator sets a <code>qps</code> (queries per seconds) value for each framework identified by its principal (You can also throttle a group of frameworks together but we’ll assume individual frameworks in this doc unless otherwise stated; see the <code>RateLimits</code> <a href="https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto">ProtoBuf definition</a> and the configuration notes below). The master then promises not to process messages from that framework at a rate above <code>qps</code>. The outstanding messages are stored in memory on the master.</p>
+
+<h2>Rate Limits Configuration</h2>
+
+<p>The following is a sample config file (in JSON format) which could be specified with the <code>--rate-limits</code> master flag.</p>
+
+<pre><code>{
+ "limits": [
+ {
+ "principal": "foo",
+ "qps": 55.5
+ "capacity": 100000
+ },
+ {
+ "principal": "bar",
+ "qps": 300
+ },
+ {
+ "principal": "baz",
+ }
+ ],
+ "aggregate_default_qps": 333,
+ "aggregate_default_capacity": 1000000
+}
+</code></pre>
+
+<p>In this example, framework <code>foo</code> is throttled at the configured <code>qps</code> and <code>capacity</code>, framework <code>bar</code> is given unlimited capacity and framework <code>baz</code> is not throttled at all. If there is a fourth framework <code>qux</code> or a framework without a principal connected to the master, it is throttled by the rules <code>aggregate_default_qps</code> and <code>aggregate_default_capacity</code>.</p>
+
+<h3>Configuration Notes</h3>
+
+<p>Below are the fields in the JSON configuration.</p>
+
+<ul>
+<li><strong>principal</strong>: (Required) uniquely identifies the entity being throttled or given unlimited rate explicitly.
+
+<ul>
+<li>It should match the framework’s <code>FrameworkInfo.principal</code> (See <a href="https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto">definition</a>).</li>
+<li>You can have multiple frameworks use the same principal (e.g., some Mesos frameworks launch a new framework instance for each job), in which case the combined traffic from all frameworks using the same principal are throttled at the specified QPS.</li>
+</ul>
+</li>
+<li><strong>qps</strong>: (Optional) queries per second, i.e., the rate.
+
+<ul>
+<li>Once set, the master guarantees that it does not process messages from this principal higher than this rate. However the master could be slower than this rate, especially if the specified rate is too high.</li>
+<li>To explicitly give a framework unlimited rate (i.e., not throttling it), add an entry to <code>limits</code> without the qps.</li>
+</ul>
+</li>
+<li><strong>capacity</strong>: (Optional) The number of <em>outstanding</em> messages frameworks of this principal can put on the master. If not specified, this principal is given unlimited capacity. Note that it is possible the queued messages use too much memory and cause the master to OOM if the capacity is set too high or not set.
+
+<ul>
+<li>NOTE: If <code>qps</code> is not specified, <code>capacity</code> is ignored.</li>
+</ul>
+</li>
+<li>Use <strong>aggregate_default_qps</strong> and <strong>aggregate_default_capacity</strong> to safeguard the master from unspecified frameworks. All the frameworks not specified in <code>limits</code> get this default rate and capacity.
+
+<ul>
+<li>The rate and capacity are aggregate values for all of them, i.e., their combined traffic is throttled together.</li>
+<li>Same as above, if <code>aggregate_default_qps</code> is not specified, <code>aggregate_default_capacity</code> is ignored.</li>
+<li>If these fields are not present, the unspecified frameworks are not throttled.
+This is an implicit way of giving frameworks unlimited rate compared to the explicit way above (using an entry in <code>limits</code> with only the principal).
+We recommend using the explicit option especially when the master does not require authentication to prevent unexpected frameworks from overwhelming the master.</li>
+</ul>
+</li>
+</ul>
+
+
+<h2>Using Framework Rate Limiting</h2>
+
+<h3>Monitoring Framework Traffic</h3>
+
+<p>While a framework is registered with the master, the master exposes counters for all messages received and processed from that framework at its metrics endpoint: <code>http://<master>/metrics/snapshot</code>. For instance, framework <code>foo</code> has two message counters <code>frameworks/foo/messages_received</code> and <code>frameworks/foo/messages_processed</code>. Without framework rate limiting the two numbers should differ by little or none (because messages are processed ASAP) but when a framework is being throttled the difference indicates the outstanding messages as a result of the throttling.</p>
+
+<p>By continuously monitoring the counters, you can derive the rate messages arrive and how fast the message queue length for the framework is growing (if it is throttled). This should depict the characteristics of the framework in terms of network traffic.</p>
+
+<h2>Configuring Rate Limits</h2>
+
+<p>Since the goal for framework rate limiting is to prevent low-SLA frameworks from using <strong>too much</strong> resources and not to model their traffic and behavior as precisely as possible, you can start by using large <code>qps</code> values to throttle them. The fact that they are throttled (regardless of the configured <code>qps</code>) is already effective in giving messages from high-SLA frameworks higher priority because they are processed ASAP.</p>
+
+<p>To calculate how much <code>capacity</code> the master can handle, you need to know the memory limit for the master process, the amount of memory it typically uses to serve similar workload without rate limiting (e.g., use <code>ps -o rss $MASTER_PID</code>) and average sizes of the framework messages (queued messages are stored as <a href="https://github.com/apache/mesos/blob/master/3rdparty/libprocess/include/process/message.hpp">serialized Protocol Buffers with a few additional fields</a>) and you should sum up all capacity values in the config.
+However since this kind of calculation is imprecise, you should start with small values that tolerate reasonable temporary framework burstiness but far from the memory limit to leave enough headroom for the master and frameworks that don’t have limited capacity.</p>
+
+<h2>Handling “Capacity Exceeded” Error</h2>
+
+<p>When a framework <strong>exceeds the capacity</strong>, a FrameworkErrorMessage is sent back to the framework which will <a href="https://github.com/apache/mesos/blob/master/src/sched/sched.cpp">abort the scheduler driver and invoke the error() callback</a>. It doesn’t kill any tasks or the scheduler itself. The framework developer can choose to restart or failover the scheduler instance to remedy the consequences of dropped messages (unless your framework doesn’t assume all messages sent to the master are processed).</p>
+
+<p>After version 0.20.0 we are going to iterate on this feature by having the master send an early alert when the message queue for this framework <strong>starts to build up</strong> (<a href="https://issues.apache.org/jira/browse/MESOS-1664">MESOS-1664</a>, consider it a “soft limit”). The scheduler can react by throttling itself (to avoid the error message) or ignoring this alert if it’s a temporary burst by design.</p>
+
+<p>Before the early alerting is implemented we <strong>don’t recommend using the rate limiting feature to throttle production frameworks</strong> for now unless you are sure about the consequences of the error message. Of course it’s OK to use it to protect production frameworks by throttling other frameworks and it doesn’t have any effect on the master if it’s not explicitly enabled.</p>
+
+ </div>
+</div>
+
+
+ <hr>
+
+ <!-- footer -->
+ <div class="footer">
+ <p>© 2012-2014 <a href="http://apache.org">The Apache Software Foundation</a>.
+ Apache Mesos, the Apache feather logo, and the Apache Mesos project logo are trademarks of The Apache Software Foundation.<p>
+ </div><!-- /footer -->
+
+ </div> <!-- /container -->
+
+ <!-- JS -->
+ <script src="//code.jquery.com/jquery-1.11.0.min.js" type="text/javascript"></script>
+ <script src="//netdna.bootstrapcdn.com/bootstrap/3.1.1/js/bootstrap.min.js" type="text/javascript"></script>
+ </body>
+</html>
\ No newline at end of file
Modified: mesos/site/publish/documentation/latest/index.html
URL: http://svn.apache.org/viewvc/mesos/site/publish/documentation/latest/index.html?rev=1620233&r1=1620232&r2=1620233&view=diff
==============================================================================
--- mesos/site/publish/documentation/latest/index.html (original)
+++ mesos/site/publish/documentation/latest/index.html Sun Aug 24 23:10:33 2014
@@ -108,11 +108,14 @@
<h2>Running Mesos</h2>
<ul>
-<li><a href="/documentation/latest/configuration/">Mesos Configuration</a> for command-line arguments</li>
-<li><a href="/documentation/latest/tools/">Tools</a> for setting up and running a Mesos cluster</li>
-<li><a href="/documentation/latest/logging-and-debugging/">Logging and Debugging</a> for viewing Mesos and framework logs.</li>
+<li><a href="/documentation/latest/configuration/">Configuration</a> for command-line arguments</li>
+<li><a href="/documentation/latest/docker-containerizer/">Docker Containerizer</a> for launching a Docker image as a Task, or as an Executor</li>
+<li><a href="/documentation/latest/framework-rate-limiting/">Framework Rate Limiting</a></li>
<li><a href="/documentation/latest/high-availability/">High Availability</a> for running multiple masters simultaneously.</li>
+<li><a href="/documentation/latest/logging-and-debugging/">Logging and Debugging</a> for viewing Mesos and framework logs.</li>
+<li><a href="/documentation/latest/network-monitoring/">Network Monitoring</a></li>
<li><a href="/documentation/latest/slave-recovery/">Slave Recovery</a> for doing seamless upgrades</li>
+<li><a href="/documentation/latest/tools/">Tools</a> for setting up and running a Mesos cluster</li>
</ul>
@@ -136,11 +139,11 @@
<h2>Contributing to Mesos</h2>
<ul>
-<li><a href="/documentation/latest/mesos-developers-guide/">Mesos Developers Guide</a> includes resources for developers contributing to Mesos and the process of submitting patches for review.</li>
-<li><a href="/documentation/latest/mesos-code-internals/">Mesos Code Internals</a> overview of the codebase and internal organization.</li>
+<li><a href="/documentation/latest/mesos-code-internals/">Code Internals</a> overview of the codebase and internal organization.</li>
<li><a href="/documentation/latest/mesos-c++-style-guide/">C++ Style Guide</a></li>
+<li><a href="/documentation/latest/mesos-developers-guide/">Developers Guide</a> includes resources for developers contributing to Mesos and the process of submitting patches for review.</li>
+<li><a href="/documentation/latest/mesos-roadmap/">Development Road Map</a></li>
<li><a href="/documentation/latest/release-guide/">Release Guide</a></li>
-<li><a href="/documentation/latest/mesos-roadmap/">Mesos Development Road Map</a></li>
</ul>
Modified: mesos/site/publish/documentation/latest/mesos-c++-style-guide/index.html
URL: http://svn.apache.org/viewvc/mesos/site/publish/documentation/latest/mesos-c%2B%2B-style-guide/index.html?rev=1620233&r1=1620232&r2=1620233&view=diff
==============================================================================
--- mesos/site/publish/documentation/latest/mesos-c++-style-guide/index.html (original)
+++ mesos/site/publish/documentation/latest/mesos-c++-style-guide/index.html Sun Aug 24 23:10:33 2014
@@ -155,17 +155,17 @@
<pre>
// 1: OK.
-allocator->resourcesUnused(frameworkId, slaveId, resources, filters);
+allocator->resourcesRecovered(frameworkId, slaveId, resources, filters);
// 2: Don't use.
-allocator->resourcesUnused(frameworkId, slaveId,
+allocator->resourcesRecovered(frameworkId, slaveId,
resources, filters);
// 3: Don't use in this case due to "jaggedness".
-allocator->resourcesUnused(frameworkId,
- slaveId,
- resources,
- filters);
+allocator->resourcesRecovered(frameworkId,
+ slaveId,
+ resources,
+ filters);
// 3: In this case, 3 is OK.
foobar(someArgument,
@@ -173,14 +173,14 @@ foobar(someArgument,
theLastArgument);
// 4: OK.
-allocator->resourcesUnused(
+allocator->resourcesRecovered(
frameworkId,
slaveId,
resources,
filters);
// 5: OK.
-allocator->resourcesUnused(
+allocator->resourcesRecovered(
frameworkId, slaveId, resources, filters);
</pre>
Added: mesos/site/publish/documentation/latest/network-monitoring/index.html
URL: http://svn.apache.org/viewvc/mesos/site/publish/documentation/latest/network-monitoring/index.html?rev=1620233&view=auto
==============================================================================
--- mesos/site/publish/documentation/latest/network-monitoring/index.html (added)
+++ mesos/site/publish/documentation/latest/network-monitoring/index.html Sun Aug 24 23:10:33 2014
@@ -0,0 +1,246 @@
+<!DOCTYPE html>
+<!--
+
+ ______ __
+ /\ _ \ /\ \
+ \ \ \L\ \ _____ __ ___\ \ \___ __
+ \ \ __ \/\ '__`\ /'__`\ /'___\ \ _ `\ /'__`\
+ \ \ \/\ \ \ \L\ \/\ \L\.\_/\ \__/\ \ \ \ \/\ __/
+ \ \_\ \_\ \ ,__/\ \__/.\_\ \____\\ \_\ \_\ \____\
+ \/_/\/_/\ \ \/ \/__/\/_/\/____/ \/_/\/_/\/____/
+ \ \_\
+ \/_/
+
+ /'\_/`\
+ /\ \ __ ____ ___ ____
+ \ \ \__\ \ /'__`\ /',__\ / __`\ /',__\
+ \ \ \_/\ \/\ __//\__, `\/\ \L\ \/\__, `\
+ \ \_\\ \_\ \____\/\____/\ \____/\/\____/
+ \/_/ \/_/\/____/\/___/ \/___/ \/___/
+
+-->
+<html>
+ <head>
+ <meta charset="utf-8">
+ <title></title>
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+ <meta name="description" content="">
+ <meta name="author" content="">
+
+ <!-- Le styles -->
+ <link href="//netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
+
+ <link href="../../../assets/css/main.css" media="screen" rel="stylesheet" type="text/css" />
+
+
+
+ <!-- Google Analytics Magic -->
+ <script type="text/javascript">
+ var _gaq = _gaq || [];
+ _gaq.push(['_setAccount', 'UA-20226872-1']);
+ _gaq.push(['_setDomainName', 'apache.org']);
+ _gaq.push(['_trackPageview']);
+
+ (function() {
+ var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
+ ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
+ var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
+ })();
+ </script>
+ </head>
+ <body>
+ <!-- magical breadcrumbs -->
+ <div class="topnav">
+ <ul class="breadcrumb">
+ <li>
+ <div class="dropdown">
+ <a data-toggle="dropdown" href="#">Apache Software Foundation <span class="caret"></span></a>
+ <ul class="dropdown-menu" role="menu" aria-labelledby="dLabel">
+ <li><a href="http://www.apache.org">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ </ul>
+ </div>
+ </li>
+ <li><a href="http://mesos.apache.org">Apache Mesos</a></li>
+
+
+ <li><a href="/documentation
+/">Documentation
+</a></li>
+
+
+ </ul><!-- /breadcrumb -->
+ </div>
+
+ <!-- navbar excitement -->
+ <div class="navbar navbar-static-top" role="navigation">
+ <div class="navbar-inner">
+ <div class="container">
+ <a href="/" class="logo"><img src="/assets/img/mesos_logo.png" alt="Apache Mesos logo" /></a>
+ <div class="nav-collapse">
+ <ul class="nav nav-pills navbar-right">
+ <li><a href="/gettingstarted/">Getting Started</a></li>
+ <li><a href="/documentation/latest/">Documentation</a></li>
+ <li><a href="/downloads/">Downloads</a></li>
+ <li><a href="/community/">Community</a></li>
+ </ul>
+ </div>
+ </div>
+ </div>
+ </div><!-- /.navbar -->
+
+ <div class="container">
+
+ <div class="row-fluid">
+ <div class="col-md-4">
+ <h4>If you're new to Mesos</h4>
+ <p>See the <a href="/gettingstarted/">getting started</a> page for more information about downloading, building, and deploying Mesos.</p>
+
+ <h4>If you'd like to get involved or you're looking for support</h4>
+ <p>See our <a href="/community/">community</a> page for more details.</p>
+ </div>
+ <div class="col-md-8">
+ <h1>Network Monitoring</h1>
+
+<p>Mesos 0.20.0 adds the support for per container network monitoring. Network statistics for each active container can be retrieved through the <code>/monitor/statistics.json</code> endpoint on the slave.</p>
+
+<p>The current solution is completely transparent to the tasks running on the slave. In other words, tasks will not notice any difference as if they were running on a slave without network monitoring turned on and were sharing the network of the slave.</p>
+
+<h2>How to setup?</h2>
+
+<p>To turn on network monitoring on your mesos cluster, you need to follow the following procedures.</p>
+
+<h3>Prerequisites</h3>
+
+<p>Currently, network monitoring is only supported on Linux. Make sure your kernel is at least 3.6. Also, check your kernel to make sure that the following upstream patches are merged in (Mesos will automatically check for those kernel functionalities and will abort if they are not supported):</p>
+
+<ul>
+<li><a href="https://github.com/torvalds/linux/commit/6a662719c9868b3d6c7d26b3a085f0cd3cc15e64">6a662719c9868b3d6c7d26b3a085f0cd3cc15e64</a></li>
+<li><a href="https://github.com/torvalds/linux/commit/0d5edc68739f1c1e0519acbea1d3f0c1882a15d7">0d5edc68739f1c1e0519acbea1d3f0c1882a15d7</a></li>
+<li><a href="https://github.com/torvalds/linux/commit/e374c618b1465f0292047a9f4c244bd71ab5f1f0">e374c618b1465f0292047a9f4c244bd71ab5f1f0</a></li>
+<li><a href="https://github.com/torvalds/linux/commit/25f929fbff0d1bcebf2e92656d33025cd330cbf8">25f929fbff0d1bcebf2e92656d33025cd330cbf8</a></li>
+</ul>
+
+
+<p>Make sure the following packages are installed on the slave:</p>
+
+<ul>
+<li><a href="http://www.infradead.org/~tgr/libnl/">libnl3</a> >= 3.2.25</li>
+<li><a href="http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2">iproute</a> (>= 2.6.39 is advised but not required for debugging purpose)</li>
+</ul>
+
+
+<p>On the build machine, you need to install the following packages:</p>
+
+<ul>
+<li><a href="http://www.infradead.org/~tgr/libnl/">libnl3-devel</a> >= 3.2.25</li>
+</ul>
+
+
+<h3>Configure and build</h3>
+
+<p>Network monitoring will NOT be built in by default. To build Mesos with network monitoring support, you need to add a configure option:</p>
+
+<pre><code>$ ./configure --with-network-isolator
+$ make
+</code></pre>
+
+<h3>Host ephemeral ports squeeze</h3>
+
+<p>With network monitoring being turned on, each container on the slave will have a separate network stack (via Linux <a href="http://lwn.net/Articles/580893/">network namespaces</a>). All containers share the same public IP of the slave (so that service discovery mechanism does not need to be changed). Each container will be assigned a subset of the ports from the host, and is only allowed to use those ports to make connections with other hosts.</p>
+
+<p>For non-ephemeral ports (e.g, listening ports), Mesos already exposes that to the scheduler (resource: ‘ports’). The scheduler is responsible for allocating those ports to executors/tasks.</p>
+
+<p>For ephemeral ports, without network monitoring, all executors/tasks running on the slave share the same ephemeral port range of the host. The default ephemeral port range on most Linux distributions is [32768, 61000]. With network monitoring, for each container, we need to reserve a range for ports on the host which will be used as the ephemeral port range for the container network stack (these ports are directly mapped into the container). We need to ensure none of the host processes are using those ports. Because of that, you may want to squeeze the host ephemeral port range in order to support more containers on each slave. To do that, you can use the following command (need root permission). A host reboot is required to ensure there are no connections using ports outside the new ephemeral range.</p>
+
+<pre><code># This sets the host ephemeral port range to [57345, 61000].
+$ echo "57345 61000" > /proc/sys/net/ipv4/ip_local_port_range
+</code></pre>
+
+<h3>Turn on network monitoring</h3>
+
+<p>After the host ephemeral ports squeeze and reboot, you can turn on network monitoring by appending <code>network/port_mapping</code> to the isolation flag. Notice that you need specify the <code>ephemeral_ports</code> resource (via –resources flag). It tells the slave which ports on the host are reserved for containers. It must NOT overlap with the host ephemeral port range. You can also specify how many ephemeral ports you want to allocate to each container. It is recommended but not required that this number is power of 2 aligned (e.g., 512, 1024). If not, there will be some performance impact for classifying packets. The maximum number of containers on the slave will be limited by approximately |ephemeral_ports|/ephemeral_ports_per_container, subject to alignment etc.</p>
+
+<pre><code>mesos-slave \
+ --checkpoint \
+ --log_dir=/var/log/mesos \
+ --work_dir=/var/lib/mesos \
+ --isolation=cgroups/cpu,cgroups/mem,network/port_mapping \
+ --resources=cpus:22;mem:62189;ports:[31000-32000];disk:400000;ephemeral_ports:[32768-57344] \
+ --ephemeral_ports_per_container=1024
+</code></pre>
+
+<h2>How to get statistics?</h2>
+
+<p>Currently, we report the following network statistics:</p>
+
+<ul>
+<li><em>net_rx_bytes</em></li>
+<li><em>net_rx_dropped</em></li>
+<li><em>net_rx_errors</em></li>
+<li><em>net_rx_packets</em></li>
+<li><em>net_tx_bytes</em></li>
+<li><em>net_tx_dropped</em></li>
+<li><em>net_tx_errors</em></li>
+<li><em>net_tx_packets</em></li>
+</ul>
+
+
+<p>For example, these are the statistics you will get by hitting the <code>/monitor/statistics.json</code> endpoint on a slave with network monitoring turned on:</p>
+
+<pre><code>$ curl -s http://localhost:5051/monitor/statistics.json | python2.6
+-mjson.tool
+[
+ {
+ "executor_id": "sample_executor_id-ebd8fa62-757d-489e-9e23-678a21d078d6",
+ "executor_name": "sample_executor",
+ "framework_id": "201103282247-0000000019-0000",
+ "source": "sample_executor",
+ "statistics": {
+ "cpus_limit": 0.35,
+ "cpus_nr_periods": 520883,
+ "cpus_nr_throttled": 2163,
+ "cpus_system_time_secs": 154.42,
+ "cpus_throttled_time_secs": 145.96,
+ "cpus_user_time_secs": 258.74,
+ "mem_anon_bytes": 109137920,
+ "mem_file_bytes": 30613504,
+ "mem_limit_bytes": 167772160,
+ "mem_mapped_file_bytes": 8192,
+ "mem_rss_bytes": 140341248,
+ "net_rx_bytes": 2402099,
+ "net_rx_dropped": 0,
+ "net_rx_errors": 0,
+ "net_rx_packets": 33273,
+ "net_tx_bytes": 1507798,
+ "net_tx_dropped": 0,
+ "net_tx_errors": 0,
+ "net_tx_packets": 17726,
+ "timestamp": 1408043826.91626
+ }
+ }
+]
+</code></pre>
+
+ </div>
+</div>
+
+
+ <hr>
+
+ <!-- footer -->
+ <div class="footer">
+ <p>© 2012-2014 <a href="http://apache.org">The Apache Software Foundation</a>.
+ Apache Mesos, the Apache feather logo, and the Apache Mesos project logo are trademarks of The Apache Software Foundation.<p>
+ </div><!-- /footer -->
+
+ </div> <!-- /container -->
+
+ <!-- JS -->
+ <script src="//code.jquery.com/jquery-1.11.0.min.js" type="text/javascript"></script>
+ <script src="//netdna.bootstrapcdn.com/bootstrap/3.1.1/js/bootstrap.min.js" type="text/javascript"></script>
+ </body>
+</html>
\ No newline at end of file
Modified: mesos/site/publish/documentation/latest/upgrades/index.html
URL: http://svn.apache.org/viewvc/mesos/site/publish/documentation/latest/upgrades/index.html?rev=1620233&r1=1620232&r2=1620233&view=diff
==============================================================================
--- mesos/site/publish/documentation/latest/upgrades/index.html (original)
+++ mesos/site/publish/documentation/latest/upgrades/index.html Sun Aug 24 23:10:33 2014
@@ -107,11 +107,66 @@
<p>This document serves as a guide for users who wish to upgrade an existing mesos cluster. Some versions require particular upgrade techniques when upgrading a running cluster. Some upgrades will have incompatible changes.</p>
+<h2>Upgrading from 0.19.x to 0.20.x.</h2>
+
+<p><strong>NOTE</strong>: The Mesos API has been changed slightly in this release. The CommandInfo has been changed (see below), which makes launching a command more flexible. The ‘value’ field has been changed from <em>required</em> to <em>optional</em>. However, it will not cause any issue during the upgrade (since the existing schedulers always set this field).</p>
+
+<pre><code>message CommandInfo {
+ ...
+ // There are two ways to specify the command:
+ // 1) If 'shell == true', the command will be launched via shell
+ // (i.e., /bin/sh -c 'value'). The 'value' specified will be
+ // treated as the shell command. The 'arguments' will be ignored.
+ // 2) If 'shell == false', the command will be launched by passing
+ // arguments to an executable. The 'value' specified will be
+ // treated as the filename of the executable. The 'arguments'
+ // will be treated as the arguments to the executable. This is
+ // similar to how POSIX exec families launch processes (i.e.,
+ // execlp(value, arguments(0), arguments(1), ...)).
+ optional bool shell = 6 [default = true];
+ optional string value = 3;
+ repeated string arguments = 7;
+ ...
+}
+</code></pre>
+
+<p><strong>NOTE</strong>: The Python bindings are also changing in this release. There are now sub-modules which allow you to use either the interfaces and/or the native driver.</p>
+
+<ul>
+<li><code>import mesos.native</code> for the native drivers</li>
+<li><code>import mesos.interface</code> for the stub implementations and protobufs</li>
+</ul>
+
+
+<p>To ensure a smooth upgrade, we recommend to upgrade your python framework and executor first. You will be able to either import using the new configuration or the old. Replace the existing imports with something like the following:</p>
+
+<pre><code> try:
+ from mesos.native import MesosExecutorDriver, MesosSchedulerDriver
+ from mesos.interface import Executor, Scheduler
+ from mesos.interface import mesos_pb2
+ except ImportError:
+ from mesos import Executor, MesosExecutorDriver, MesosSchedulerDriver, Scheduler
+ import mesos_pb2
+</code></pre>
+
+<p><strong>NOTE</strong>: If you’re using a pure language binding, please ensure that it sends status update acknowledgements through the master before upgrading.</p>
+
+<p>In order to upgrade a running cluster:</p>
+
+<ul>
+<li>Install the new master binaries and restart the masters.</li>
+<li>Install the new slave binaries and restart the slaves.</li>
+<li>Upgrade the schedulers by linking the latest native library (install the latest mesos jar and python egg if necessary).</li>
+<li>Restart the schedulers.</li>
+<li>Upgrade the executors by linking the latest native library (install the latest mesos jar and python egg if necessary).</li>
+</ul>
+
+
<h2>Upgrading from 0.18.x to 0.19.x.</h2>
-<p>NOTE: There are new required flags on the master (<code>--work_dir</code> and <code>--quorum</code>) to support the <em>Registrar</em> feature, which adds replicated state on the masters.</p>
+<p><strong>NOTE</strong>: There are new required flags on the master (<code>--work_dir</code> and <code>--quorum</code>) to support the <em>Registrar</em> feature, which adds replicated state on the masters.</p>
-<p>NOTE: No required upgrade ordering across components.</p>
+<p><strong>NOTE</strong>: No required upgrade ordering across components.</p>
<p>In order to upgrade a running cluster:</p>
@@ -128,7 +183,7 @@
<p>In order to upgrade a running cluster:</p>
-<p>Note: This upgrade requires a system reboot for slaves that use Linux cgroups for isolation.</p>
+<p><strong>NOTE</strong>: This upgrade requires a system reboot for slaves that use Linux cgroups for isolation.</p>
<ul>
<li>Install the new master binaries and restart the masters.</li>
Modified: mesos/site/publish/documentation/mesos-c++-style-guide/index.html
URL: http://svn.apache.org/viewvc/mesos/site/publish/documentation/mesos-c%2B%2B-style-guide/index.html?rev=1620233&r1=1620232&r2=1620233&view=diff
==============================================================================
--- mesos/site/publish/documentation/mesos-c++-style-guide/index.html (original)
+++ mesos/site/publish/documentation/mesos-c++-style-guide/index.html Sun Aug 24 23:10:33 2014
@@ -155,17 +155,17 @@
<pre>
// 1: OK.
-allocator->resourcesUnused(frameworkId, slaveId, resources, filters);
+allocator->resourcesRecovered(frameworkId, slaveId, resources, filters);
// 2: Don't use.
-allocator->resourcesUnused(frameworkId, slaveId,
+allocator->resourcesRecovered(frameworkId, slaveId,
resources, filters);
// 3: Don't use in this case due to "jaggedness".
-allocator->resourcesUnused(frameworkId,
- slaveId,
- resources,
- filters);
+allocator->resourcesRecovered(frameworkId,
+ slaveId,
+ resources,
+ filters);
// 3: In this case, 3 is OK.
foobar(someArgument,
@@ -173,14 +173,14 @@ foobar(someArgument,
theLastArgument);
// 4: OK.
-allocator->resourcesUnused(
+allocator->resourcesRecovered(
frameworkId,
slaveId,
resources,
filters);
// 5: OK.
-allocator->resourcesUnused(
+allocator->resourcesRecovered(
frameworkId, slaveId, resources, filters);
</pre>
Added: mesos/site/publish/documentation/network-monitoring/index.html
URL: http://svn.apache.org/viewvc/mesos/site/publish/documentation/network-monitoring/index.html?rev=1620233&view=auto
==============================================================================
--- mesos/site/publish/documentation/network-monitoring/index.html (added)
+++ mesos/site/publish/documentation/network-monitoring/index.html Sun Aug 24 23:10:33 2014
@@ -0,0 +1,246 @@
+<!DOCTYPE html>
+<!--
+
+ ______ __
+ /\ _ \ /\ \
+ \ \ \L\ \ _____ __ ___\ \ \___ __
+ \ \ __ \/\ '__`\ /'__`\ /'___\ \ _ `\ /'__`\
+ \ \ \/\ \ \ \L\ \/\ \L\.\_/\ \__/\ \ \ \ \/\ __/
+ \ \_\ \_\ \ ,__/\ \__/.\_\ \____\\ \_\ \_\ \____\
+ \/_/\/_/\ \ \/ \/__/\/_/\/____/ \/_/\/_/\/____/
+ \ \_\
+ \/_/
+
+ /'\_/`\
+ /\ \ __ ____ ___ ____
+ \ \ \__\ \ /'__`\ /',__\ / __`\ /',__\
+ \ \ \_/\ \/\ __//\__, `\/\ \L\ \/\__, `\
+ \ \_\\ \_\ \____\/\____/\ \____/\/\____/
+ \/_/ \/_/\/____/\/___/ \/___/ \/___/
+
+-->
+<html>
+ <head>
+ <meta charset="utf-8">
+ <title></title>
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+ <meta name="description" content="">
+ <meta name="author" content="">
+
+ <!-- Le styles -->
+ <link href="//netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
+
+ <link href="../../assets/css/main.css" media="screen" rel="stylesheet" type="text/css" />
+
+
+
+ <!-- Google Analytics Magic -->
+ <script type="text/javascript">
+ var _gaq = _gaq || [];
+ _gaq.push(['_setAccount', 'UA-20226872-1']);
+ _gaq.push(['_setDomainName', 'apache.org']);
+ _gaq.push(['_trackPageview']);
+
+ (function() {
+ var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
+ ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
+ var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
+ })();
+ </script>
+ </head>
+ <body>
+ <!-- magical breadcrumbs -->
+ <div class="topnav">
+ <ul class="breadcrumb">
+ <li>
+ <div class="dropdown">
+ <a data-toggle="dropdown" href="#">Apache Software Foundation <span class="caret"></span></a>
+ <ul class="dropdown-menu" role="menu" aria-labelledby="dLabel">
+ <li><a href="http://www.apache.org">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ </ul>
+ </div>
+ </li>
+ <li><a href="http://mesos.apache.org">Apache Mesos</a></li>
+
+
+ <li><a href="/documentation
+/">Documentation
+</a></li>
+
+
+ </ul><!-- /breadcrumb -->
+ </div>
+
+ <!-- navbar excitement -->
+ <div class="navbar navbar-static-top" role="navigation">
+ <div class="navbar-inner">
+ <div class="container">
+ <a href="/" class="logo"><img src="/assets/img/mesos_logo.png" alt="Apache Mesos logo" /></a>
+ <div class="nav-collapse">
+ <ul class="nav nav-pills navbar-right">
+ <li><a href="/gettingstarted/">Getting Started</a></li>
+ <li><a href="/documentation/latest/">Documentation</a></li>
+ <li><a href="/downloads/">Downloads</a></li>
+ <li><a href="/community/">Community</a></li>
+ </ul>
+ </div>
+ </div>
+ </div>
+ </div><!-- /.navbar -->
+
+ <div class="container">
+
+ <div class="row-fluid">
+ <div class="col-md-4">
+ <h4>If you're new to Mesos</h4>
+ <p>See the <a href="/gettingstarted/">getting started</a> page for more information about downloading, building, and deploying Mesos.</p>
+
+ <h4>If you'd like to get involved or you're looking for support</h4>
+ <p>See our <a href="/community/">community</a> page for more details.</p>
+ </div>
+ <div class="col-md-8">
+ <h1>Network Monitoring</h1>
+
+<p>Mesos 0.20.0 adds the support for per container network monitoring. Network statistics for each active container can be retrieved through the <code>/monitor/statistics.json</code> endpoint on the slave.</p>
+
+<p>The current solution is completely transparent to the tasks running on the slave. In other words, tasks will not notice any difference as if they were running on a slave without network monitoring turned on and were sharing the network of the slave.</p>
+
+<h2>How to setup?</h2>
+
+<p>To turn on network monitoring on your mesos cluster, you need to follow the following procedures.</p>
+
+<h3>Prerequisites</h3>
+
+<p>Currently, network monitoring is only supported on Linux. Make sure your kernel is at least 3.6. Also, check your kernel to make sure that the following upstream patches are merged in (Mesos will automatically check for those kernel functionalities and will abort if they are not supported):</p>
+
+<ul>
+<li><a href="https://github.com/torvalds/linux/commit/6a662719c9868b3d6c7d26b3a085f0cd3cc15e64">6a662719c9868b3d6c7d26b3a085f0cd3cc15e64</a></li>
+<li><a href="https://github.com/torvalds/linux/commit/0d5edc68739f1c1e0519acbea1d3f0c1882a15d7">0d5edc68739f1c1e0519acbea1d3f0c1882a15d7</a></li>
+<li><a href="https://github.com/torvalds/linux/commit/e374c618b1465f0292047a9f4c244bd71ab5f1f0">e374c618b1465f0292047a9f4c244bd71ab5f1f0</a></li>
+<li><a href="https://github.com/torvalds/linux/commit/25f929fbff0d1bcebf2e92656d33025cd330cbf8">25f929fbff0d1bcebf2e92656d33025cd330cbf8</a></li>
+</ul>
+
+
+<p>Make sure the following packages are installed on the slave:</p>
+
+<ul>
+<li><a href="http://www.infradead.org/~tgr/libnl/">libnl3</a> >= 3.2.25</li>
+<li><a href="http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2">iproute</a> (>= 2.6.39 is advised but not required for debugging purpose)</li>
+</ul>
+
+
+<p>On the build machine, you need to install the following packages:</p>
+
+<ul>
+<li><a href="http://www.infradead.org/~tgr/libnl/">libnl3-devel</a> >= 3.2.25</li>
+</ul>
+
+
+<h3>Configure and build</h3>
+
+<p>Network monitoring will NOT be built in by default. To build Mesos with network monitoring support, you need to add a configure option:</p>
+
+<pre><code>$ ./configure --with-network-isolator
+$ make
+</code></pre>
+
+<h3>Host ephemeral ports squeeze</h3>
+
+<p>With network monitoring being turned on, each container on the slave will have a separate network stack (via Linux <a href="http://lwn.net/Articles/580893/">network namespaces</a>). All containers share the same public IP of the slave (so that service discovery mechanism does not need to be changed). Each container will be assigned a subset of the ports from the host, and is only allowed to use those ports to make connections with other hosts.</p>
+
+<p>For non-ephemeral ports (e.g, listening ports), Mesos already exposes that to the scheduler (resource: ‘ports’). The scheduler is responsible for allocating those ports to executors/tasks.</p>
+
+<p>For ephemeral ports, without network monitoring, all executors/tasks running on the slave share the same ephemeral port range of the host. The default ephemeral port range on most Linux distributions is [32768, 61000]. With network monitoring, for each container, we need to reserve a range for ports on the host which will be used as the ephemeral port range for the container network stack (these ports are directly mapped into the container). We need to ensure none of the host processes are using those ports. Because of that, you may want to squeeze the host ephemeral port range in order to support more containers on each slave. To do that, you can use the following command (need root permission). A host reboot is required to ensure there are no connections using ports outside the new ephemeral range.</p>
+
+<pre><code># This sets the host ephemeral port range to [57345, 61000].
+$ echo "57345 61000" > /proc/sys/net/ipv4/ip_local_port_range
+</code></pre>
+
+<h3>Turn on network monitoring</h3>
+
+<p>After the host ephemeral ports squeeze and reboot, you can turn on network monitoring by appending <code>network/port_mapping</code> to the isolation flag. Notice that you need specify the <code>ephemeral_ports</code> resource (via –resources flag). It tells the slave which ports on the host are reserved for containers. It must NOT overlap with the host ephemeral port range. You can also specify how many ephemeral ports you want to allocate to each container. It is recommended but not required that this number is power of 2 aligned (e.g., 512, 1024). If not, there will be some performance impact for classifying packets. The maximum number of containers on the slave will be limited by approximately |ephemeral_ports|/ephemeral_ports_per_container, subject to alignment etc.</p>
+
+<pre><code>mesos-slave \
+ --checkpoint \
+ --log_dir=/var/log/mesos \
+ --work_dir=/var/lib/mesos \
+ --isolation=cgroups/cpu,cgroups/mem,network/port_mapping \
+ --resources=cpus:22;mem:62189;ports:[31000-32000];disk:400000;ephemeral_ports:[32768-57344] \
+ --ephemeral_ports_per_container=1024
+</code></pre>
+
+<h2>How to get statistics?</h2>
+
+<p>Currently, we report the following network statistics:</p>
+
+<ul>
+<li><em>net_rx_bytes</em></li>
+<li><em>net_rx_dropped</em></li>
+<li><em>net_rx_errors</em></li>
+<li><em>net_rx_packets</em></li>
+<li><em>net_tx_bytes</em></li>
+<li><em>net_tx_dropped</em></li>
+<li><em>net_tx_errors</em></li>
+<li><em>net_tx_packets</em></li>
+</ul>
+
+
+<p>For example, these are the statistics you will get by hitting the <code>/monitor/statistics.json</code> endpoint on a slave with network monitoring turned on:</p>
+
+<pre><code>$ curl -s http://localhost:5051/monitor/statistics.json | python2.6
+-mjson.tool
+[
+ {
+ "executor_id": "sample_executor_id-ebd8fa62-757d-489e-9e23-678a21d078d6",
+ "executor_name": "sample_executor",
+ "framework_id": "201103282247-0000000019-0000",
+ "source": "sample_executor",
+ "statistics": {
+ "cpus_limit": 0.35,
+ "cpus_nr_periods": 520883,
+ "cpus_nr_throttled": 2163,
+ "cpus_system_time_secs": 154.42,
+ "cpus_throttled_time_secs": 145.96,
+ "cpus_user_time_secs": 258.74,
+ "mem_anon_bytes": 109137920,
+ "mem_file_bytes": 30613504,
+ "mem_limit_bytes": 167772160,
+ "mem_mapped_file_bytes": 8192,
+ "mem_rss_bytes": 140341248,
+ "net_rx_bytes": 2402099,
+ "net_rx_dropped": 0,
+ "net_rx_errors": 0,
+ "net_rx_packets": 33273,
+ "net_tx_bytes": 1507798,
+ "net_tx_dropped": 0,
+ "net_tx_errors": 0,
+ "net_tx_packets": 17726,
+ "timestamp": 1408043826.91626
+ }
+ }
+]
+</code></pre>
+
+ </div>
+</div>
+
+
+ <hr>
+
+ <!-- footer -->
+ <div class="footer">
+ <p>© 2012-2014 <a href="http://apache.org">The Apache Software Foundation</a>.
+ Apache Mesos, the Apache feather logo, and the Apache Mesos project logo are trademarks of The Apache Software Foundation.<p>
+ </div><!-- /footer -->
+
+ </div> <!-- /container -->
+
+ <!-- JS -->
+ <script src="//code.jquery.com/jquery-1.11.0.min.js" type="text/javascript"></script>
+ <script src="//netdna.bootstrapcdn.com/bootstrap/3.1.1/js/bootstrap.min.js" type="text/javascript"></script>
+ </body>
+</html>
\ No newline at end of file
Modified: mesos/site/publish/documentation/upgrades/index.html
URL: http://svn.apache.org/viewvc/mesos/site/publish/documentation/upgrades/index.html?rev=1620233&r1=1620232&r2=1620233&view=diff
==============================================================================
--- mesos/site/publish/documentation/upgrades/index.html (original)
+++ mesos/site/publish/documentation/upgrades/index.html Sun Aug 24 23:10:33 2014
@@ -107,11 +107,66 @@
<p>This document serves as a guide for users who wish to upgrade an existing mesos cluster. Some versions require particular upgrade techniques when upgrading a running cluster. Some upgrades will have incompatible changes.</p>
+<h2>Upgrading from 0.19.x to 0.20.x.</h2>
+
+<p><strong>NOTE</strong>: The Mesos API has been changed slightly in this release. The CommandInfo has been changed (see below), which makes launching a command more flexible. The ‘value’ field has been changed from <em>required</em> to <em>optional</em>. However, it will not cause any issue during the upgrade (since the existing schedulers always set this field).</p>
+
+<pre><code>message CommandInfo {
+ ...
+ // There are two ways to specify the command:
+ // 1) If 'shell == true', the command will be launched via shell
+ // (i.e., /bin/sh -c 'value'). The 'value' specified will be
+ // treated as the shell command. The 'arguments' will be ignored.
+ // 2) If 'shell == false', the command will be launched by passing
+ // arguments to an executable. The 'value' specified will be
+ // treated as the filename of the executable. The 'arguments'
+ // will be treated as the arguments to the executable. This is
+ // similar to how POSIX exec families launch processes (i.e.,
+ // execlp(value, arguments(0), arguments(1), ...)).
+ optional bool shell = 6 [default = true];
+ optional string value = 3;
+ repeated string arguments = 7;
+ ...
+}
+</code></pre>
+
+<p><strong>NOTE</strong>: The Python bindings are also changing in this release. There are now sub-modules which allow you to use either the interfaces and/or the native driver.</p>
+
+<ul>
+<li><code>import mesos.native</code> for the native drivers</li>
+<li><code>import mesos.interface</code> for the stub implementations and protobufs</li>
+</ul>
+
+
+<p>To ensure a smooth upgrade, we recommend to upgrade your python framework and executor first. You will be able to either import using the new configuration or the old. Replace the existing imports with something like the following:</p>
+
+<pre><code> try:
+ from mesos.native import MesosExecutorDriver, MesosSchedulerDriver
+ from mesos.interface import Executor, Scheduler
+ from mesos.interface import mesos_pb2
+ except ImportError:
+ from mesos import Executor, MesosExecutorDriver, MesosSchedulerDriver, Scheduler
+ import mesos_pb2
+</code></pre>
+
+<p><strong>NOTE</strong>: If you’re using a pure language binding, please ensure that it sends status update acknowledgements through the master before upgrading.</p>
+
+<p>In order to upgrade a running cluster:</p>
+
+<ul>
+<li>Install the new master binaries and restart the masters.</li>
+<li>Install the new slave binaries and restart the slaves.</li>
+<li>Upgrade the schedulers by linking the latest native library (install the latest mesos jar and python egg if necessary).</li>
+<li>Restart the schedulers.</li>
+<li>Upgrade the executors by linking the latest native library (install the latest mesos jar and python egg if necessary).</li>
+</ul>
+
+
<h2>Upgrading from 0.18.x to 0.19.x.</h2>
-<p>NOTE: There are new required flags on the master (<code>--work_dir</code> and <code>--quorum</code>) to support the <em>Registrar</em> feature, which adds replicated state on the masters.</p>
+<p><strong>NOTE</strong>: There are new required flags on the master (<code>--work_dir</code> and <code>--quorum</code>) to support the <em>Registrar</em> feature, which adds replicated state on the masters.</p>
-<p>NOTE: No required upgrade ordering across components.</p>
+<p><strong>NOTE</strong>: No required upgrade ordering across components.</p>
<p>In order to upgrade a running cluster:</p>
@@ -128,7 +183,7 @@
<p>In order to upgrade a running cluster:</p>
-<p>Note: This upgrade requires a system reboot for slaves that use Linux cgroups for isolation.</p>
+<p><strong>NOTE</strong>: This upgrade requires a system reboot for slaves that use Linux cgroups for isolation.</p>
<ul>
<li>Install the new master binaries and restart the masters.</li>