You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mesos.apache.org by dl...@apache.org on 2015/03/09 18:01:57 UTC
svn commit: r1665304 [3/3] - in /mesos/site: publish/
publish/documentation/app-framework-development-guide/
publish/documentation/configuration/
publish/documentation/latest/app-framework-development-guide/
publish/documentation/latest/configuration/ ...
Modified: mesos/site/source/documentation/latest/app-framework-development-guide.md
URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/app-framework-development-guide.md?rev=1665304&r1=1665303&r2=1665304&view=diff
==============================================================================
--- mesos/site/source/documentation/latest/app-framework-development-guide.md (original)
+++ mesos/site/source/documentation/latest/app-framework-development-guide.md Mon Mar 9 17:01:57 2015
@@ -209,3 +209,43 @@ Declared in `MESOS_HOME/include/mesos/ex
You need to put your framework somewhere that all slaves on the cluster can get it from. If you are running HDFS, you can put your executor into HDFS. Then, you tell Mesos where it is via the `ExecutorInfo` parameter of `MesosSchedulerDriver`'s constructor (e.g. see src/examples/java/TestFramework.java for an example of this). ExecutorInfo is a a Protocol Buffer Message class (defined in `include/mesos/mesos.proto`), and you set its URI field to something like "HDFS://path/to/executor/". Also, you can pass the `frameworks_home` configuration option (defaults to: `MESOS_HOME/frameworks`) to your `mesos-slave` daemons when you launch them to specify where all of your framework executors are stored (e.g. on an NFS mount that is available to all slaves), then set `ExecutorInfo` to be a relative path, and the slave will prepend the value of frameworks_home to the relative path provided.
Once you are sure that your executors are available to the mesos-slaves, you should be able to run your scheduler, which will register with the Mesos master, and start receiving resource offers!
+
+## Service discovery
+
+When your framework registers an executor or launches a task, it can provide additional information for service discovery. This information is stored by the Mesos master along with other imporant information such as the slave currently running the task. A service discovery system can programmatically retrieve this information in order to set up DNS entries, configure proxies, or update any consistent store used for service discovery in a Mesos cluster that runs multiple frameworks and multiple tasks.
+
+The optional `DiscoveryInfo` message for `TaskInfo` and `ExecutorInfo` is declared in `MESOS_HOME/include/mesos/mesos.proto`
+
+```
+message DiscoveryInfo {
+ enum Visibility {
+ FRAMEWORK = 0;
+ CLUSTER = 1;
+ EXTERNAL = 2;
+ }
+
+ required Visibility visibility = 1;
+ optional string name = 2;
+ optional string environment = 3;
+ optional string location = 4;
+ optional string version = 5;
+ optional Ports ports = 6;
+ optional Labels labels = 7;
+}
+```
+
+`Visibility` is the key parameter that instructs the service discovery system whether a service should be discoverable. We currently differentiate between three cases:
+
+ - a task should not be discoverable for anyone but its framework.
+ - a task should be discoverable for all frameworks running on the Mesos cluster but not externally.
+ - a task should be made discoverable broadly.
+
+Many service discovery systems provide additional features that manage the visibility of services (e.g., ACLs in proxy based systems, security extensions to DNS, VLAN or subnet selection). It is not the intended use of the visibility field to manage such features. When a service discovery system retrieves the task or executor information from the master, it can decide how to handle tasks without DiscoveryInfo. For instance, tasks may be made non discoverable to other frameworks (equivalent to `visibility=FRAMEWORK`) or discoverable to all frameworks (equivalent to `visibility=CLUSTER`).
+
+The `name` field is a string that that provides the service discovery system with the name under which the task is discoverable. The typical use of the name field will be to provide a valid hostname. If name is not provided, it is up to the service discovery system to create a name for the task based on the name field in `taskInfo` or other information.
+
+The `environment`, `location`, and `version` fields provide first class support for common attributes used to differentiate between similar services in large deployments. The `environment` may receive values such as `PROD/QA/DEV`, the `location` field may receive values like `EAST-US/WEST-US/EUROPE/AMEA`, and the `version` field may receive values like v2.0/v0.9. The exact use of these fields is up to the service discovery system.
+
+The `ports` field allows the framework to identify the ports a task listens to and explicitly name the functionality they represent and the layer-4 protocol they use (TCP, UDP, or other). For example, a Cassandra task will define ports like `â7000,Cluster,TCPâ`, `â7001,SSL,TCPâ`, `â9160,Thrift,TCPâ`, `â9042,Native,TCPâ`, and `â7199,JMX,TCPâ`. It is up to the service discovery system to use these names and protocol in appropriate ways, potentially combining them with the `name` field in DiscoveryInfo.
+
+The `labels` field allows a framework to pass arbitrary labels to the service discovery system in the form of key/value pairs. Note that anything passed through this field is not guaranteed to be supported moving forward. Nevertheless, this field provides extensibility. Common uses of this field will allow us to identify use cases that require first class support.
Modified: mesos/site/source/documentation/latest/configuration.md
URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/configuration.md?rev=1665304&r1=1665303&r2=1665304&view=diff
==============================================================================
--- mesos/site/source/documentation/latest/configuration.md (original)
+++ mesos/site/source/documentation/latest/configuration.md Mon Mar 9 17:01:57 2015
@@ -51,16 +51,6 @@ If you have special compilation requirem
</tr>
<tr>
<td>
- --[no-]initialize_driver_logging
- </td>
- <td>
- Whether to automatically initialize google logging of scheduler
- and/or executor drivers. (default: true)
-
- </td>
- </tr>
- <tr>
- <td>
--log_dir=VALUE
</td>
<td>
@@ -138,7 +128,9 @@ If you have special compilation requirem
The size of the quorum of replicas when using 'replicated_log' based
registry. It is imperative to set this value to be a majority of
masters i.e., quorum > (number of masters)/2.
+ <p/>
+ <b>NOTE</b> Not required if master is run in standalone mode (non-HA).
</td>
</tr>
<tr>
@@ -158,7 +150,10 @@ If you have special compilation requirem
ZooKeeper URL (used for leader election amongst masters)
May be one of:
<pre><code>zk://host1:port1,host2:port2,.../path
-zk://username:password@host1:port1,host2:port2,.../path</code></pre>
+zk://username:password@host1:port1,host2:port2,.../path
+file:///path/to/file (where file contains one of the above)</code></pre>
+ <p/>
+ <b>NOTE</b> Not required if master is run in standalone mode (non-HA).
</td>
</tr>
</table>
@@ -181,8 +176,8 @@ zk://username:password@host1:port1,host2
</td>
<td>
The value is a JSON formatted string of ACLs. Remember you can also use
- the <code>file:///path/to/file</code> argument value format to write the
- JSON in a file.
+ the <code>file:///path/to/file</code> or <code>/path/to/file</code>
+ argument value format to write the JSON in a file.
<p/>
See the ACLs protobuf in mesos.proto for the expected format.
<p/>
@@ -245,7 +240,8 @@ zk://username:password@host1:port1,host2
<td>
Authenticator implementation to use when authenticating frameworks
and/or slaves. Use the default <code>crammd5</code>, or
- load an alternate authenticator module using <code>--modules</code>. (default: crammd5)
+ load an alternate authenticator module using <code>--modules</code>.
+ (default: crammd5)
</td>
</tr>
<tr>
@@ -265,7 +261,8 @@ zk://username:password@host1:port1,host2
Either a path to a text file with a list of credentials,
each line containing 'principal' and 'secret' separated by whitespace,
or, a path to a JSON-formatted file containing credentials.
- Path should be of the form <code>file:///path/to/file</code>
+ Path should be of the form <code>file:///path/to/file</code> or
+ <code>/path/to/file</code>
<p/>
JSON file Example:
<pre><code>{
@@ -285,6 +282,16 @@ zk://username:password@host1:port1,host2
</tr>
<tr>
<td>
+ --external_log_file=VALUE
+ </td>
+ <td>
+ Specified the externally managed log file. This file will be
+ exposed in the webui and HTTP api. This is useful when using
+ stderr logging as the log file is otherwise unknown to Mesos.
+ </td>
+ </tr>
+ <tr>
+ <td>
--framework_sorter=VALUE
</td>
<td>
@@ -295,6 +302,15 @@ zk://username:password@host1:port1,host2
</tr>
<tr>
<td>
+ --hooks=VALUE
+ </td>
+ <td>
+ A comma separated list of hook modules to be
+ installed inside master.
+ </td>
+ </tr>
+ <tr>
+ <td>
--hostname=VALUE
</td>
<td>
@@ -322,10 +338,10 @@ zk://username:password@host1:port1,host2
subsystems.
<p/>
Use <code>--modules=filepath</code> to specify the list of modules via a
- file containing a JSON formatted string. Remember you can also use
- the <code>file:///path/to/file</code> argument value format to write the
- JSON in a file.
- <p/>
+ file containing a JSON formatted string.
+ Remember you can also use the <code>file:///path/to/file</code> or
+ <code>/path/to/file</code> argument value format to write the JSON in a
+ file.<p/>
Use <code>--modules="{...}"</code> to specify the list of modules inline.
<p/>
JSON file example:
@@ -382,8 +398,8 @@ zk://username:password@host1:port1,host2
for framework rate limiting.
<p/>
Remember you can also use
- the <code>file:///path/to/file</code> argument value format to write the
- JSON in a file.
+ the <code>file:///path/to/file</code> or <code>/path/to/file</code>
+ argument value format to write the JSON in a file.
<p/>
See the RateLimits protobuf in mesos.proto for the expected format.
@@ -617,13 +633,11 @@ zk://username:password@host1:port1,host2
</code></pre>
</li>
- <li> a path to a file containing either one of the above options </li>
- <pre><code>You can also use the <code>file:///path/to/file</code> syntax
- to read the argument from a file which contains one of the above</code></pre>
+ <li> a path to a file containing either one of the above options. You
+ can also use the <code>file:///path/to/file</code> syntax to read the
+ argument from a file which contains one of the above.
</li>
</ol>
- Examples:
-
</td>
</tr>
</table>
@@ -652,6 +666,16 @@ zk://username:password@host1:port1,host2
</tr>
<tr>
<td>
+ --authenticatee=VALUE
+ </td>
+ <td>
+ Authenticatee implementation to use when authenticating against the
+ master. Use the default <code>crammd5</code>, or
+ load an alternate authenticatee module using <code>--modules</code>. (default: crammd5)
+ </td>
+ </tr>
+ <tr>
+ <td>
--[no-]cgroups_enable_cfs
</td>
<td>
@@ -690,14 +714,11 @@ zk://username:password@host1:port1,host2
</tr>
<tr>
<td>
- --[no-]checkpoint
+ --container_disk_watch_interval=VALUE
</td>
<td>
- This flag is deprecated and will be removed in a future release.
- Whether to checkpoint slave and frameworks information
- to disk. This enables a restarted slave to recover
- status updates and reconnect with (--recover=reconnect) or
- kill (--recover=cleanup) old executors (default: true)
+ The interval between disk quota checks for containers. This flag is
+ used for the <code>posix/disk</code> isolator. (default: 15secs)
</td>
</tr>
<tr>
@@ -785,24 +806,6 @@ zk://username:password@host1:port1,host2
</tr>
<tr>
<td>
- --docker_stop_timeout=VALUE
- </td>
- <td>
- The time as a duration for docker to wait after stopping an instance
- before it kills that instance. (default: 0secs)
- </td>
- </tr>
- <tr>
- <td>
- --network_enable_socket_statistics
- </td>
- <td>
- Whether to collect socket statistics (e.g., TCP RTT) for
- each container. (default: false)
- </td>
- </tr>
- <tr>
- <td>
--default_role=VALUE
</td>
<td>
@@ -854,6 +857,24 @@ zk://username:password@host1:port1,host2
</tr>
<tr>
<td>
+ --docker_stop_timeout=VALUE
+ </td>
+ <td>
+ The time as a duration for docker to wait after stopping an instance
+ before it kills that instance. (default: 0secs)
+ </td>
+ </tr>
+ <tr>
+ <td>
+ --[no-]enforce_container_disk_quota
+ </td>
+ <td>
+ Whether to enable disk quota enforcement for containers. This flag
+ is used for the 'posix/disk' isolator. (default: false)
+ </td>
+ </tr>
+ <tr>
+ <td>
--executor_registration_timeout=VALUE
</td>
<td>
@@ -873,6 +894,16 @@ zk://username:password@host1:port1,host2
</tr>
<tr>
<td>
+ --external_log_file=VALUE
+ </td>
+ <td>
+ Specified the externally managed log file. This file will be
+ exposed in the webui and HTTP api. This is useful when using
+ stderr logging as the log file is otherwise unknown to Mesos.
+ </td>
+ </tr>
+ <tr>
+ <td>
--frameworks_home=VALUE
</td>
<td>
@@ -893,6 +924,19 @@ zk://username:password@host1:port1,host2
</tr>
<tr>
<td>
+ --gc_disk_headroom=VALUE
+ </td>
+ <td>
+ Adjust disk headroom used to calculate maximum executor
+ directory age. Age is calculated by:</p>
+ <code>gc_delay * max(0.0, (1.0 - gc_disk_headroom - disk usage))</code>
+ every <code>--disk_watch_interval</code> duration.
+ <code>gc_disk_headroom</code> must be a value between 0.0 and 1.0
+ (default: 0.1)
+ </td>
+ </tr>
+ <tr>
+ <td>
--hadoop_home=VALUE
</td>
<td>
@@ -904,6 +948,15 @@ zk://username:password@host1:port1,host2
</tr>
<tr>
<td>
+ --hooks=VALUE
+ </td>
+ <td>
+ A comma separated list of hook modules to be
+ installed inside master.
+ </td>
+ </tr>
+ <tr>
+ <td>
--hostname=VALUE
</td>
<td>
@@ -941,9 +994,9 @@ zk://username:password@host1:port1,host2
List of modules to be loaded and be available to the internal
subsystems.
<p/>
- Remember you can also use the <code>file:///path/to/file</code>
- argument value format to have the value read from a file.
- <p/>
+ Remember you can also use the <code>file:///path/to/file</code> or
+ <code>/path/to/file</code> argument value format to have the value read
+ from a file.<p/>
Use <code>--modules="{...}"</code> to specify the list of modules inline.
<p/>
JSON file example:
Modified: mesos/site/source/documentation/latest/mesos-containerizer.md
URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/mesos-containerizer.md?rev=1665304&r1=1665303&r2=1665304&view=diff
==============================================================================
--- mesos/site/source/documentation/latest/mesos-containerizer.md (original)
+++ mesos/site/source/documentation/latest/mesos-containerizer.md Mon Mar 9 17:01:57 2015
@@ -58,3 +58,25 @@ to freezing cgroups under OOM conditions
/proc will be mounted for containers so tools such as 'ps' will work
correctly.
+
+
+### Posix Disk Isolator
+
+The Posix Disk isolator provides basic disk isolation. It is able to
+report the disk usage for each sandbox and optionally enforce the disk
+quota. It can be used on both Linux and OS X.
+
+To enable the Posix Disk isolator, append `posix/disk` to the
+`--isolation` flag when starting the slave.
+
+By default, the disk quota enforcement is disabled. To enable it,
+specify `--enforce_container_disk_quota` when starting the slave.
+
+The Posix Disk isolator reports disk usage for each sandbox by
+periodically running the `du` command. The disk usage can be retrieved
+from the resource statistics endpoint (`/monitor/statistics.json`).
+
+The interval between two `du`s can be controlled by the slave flag
+`--container_disk_watch_interval`. For example,
+`--container_disk_watch_interval=1mins` sets the interval to be 1
+minute. The default interval is 15 seconds.
Modified: mesos/site/source/documentation/latest/mesos-presentations.md
URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/mesos-presentations.md?rev=1665304&r1=1665303&r2=1665304&view=diff
==============================================================================
--- mesos/site/source/documentation/latest/mesos-presentations.md (original)
+++ mesos/site/source/documentation/latest/mesos-presentations.md Mon Mar 9 17:01:57 2015
@@ -5,6 +5,11 @@ layout: documentation
# Video and Slides of Mesos Presentations
_(Listed in date descending order)_
+## Building and Deploying Applications to Apache Mesos
+([Slides](https://www.slideshare.net/charmalloc/buildingdeployingapplicationsmesos))
+Joe Stein
+Presented February 26, 2015 at [DigitalOcean Community Meetup](http://www.meetup.com/DigitalOcean_Community/events/220580767/)
+
## MesosCon 2014
[Video playlist](https://www.youtube.com/playlist?list=PLDVc2EaAVPg9kp8cFzjR1Yxj96I4U5EGN)
@@ -42,7 +47,7 @@ Presented April 3rd, 2014 at @TwitterOSS
Paco Nathan
Presented February 13th, 2014 at [Strata](http://strataconf.com/)
-### Run your data center like Google's with Apache Mesos ([Video + Demo](https://www.youtube.com/watch?v=2YWVGMuMTrg))
+### Run your data center like Google's with Apache Mesos ([Video + Demo](https://www.youtube.com/watch?v=2YWVGMuMTrg))
Abhishek Parolkar
Presented November 14th, 2013 at [Cloud Expo Asia 2013](http://www.cloudexpoasia.com/)
Modified: mesos/site/source/documentation/latest/powered-by-mesos.md
URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/powered-by-mesos.md?rev=1665304&r1=1665303&r2=1665304&view=diff
==============================================================================
--- mesos/site/source/documentation/latest/powered-by-mesos.md (original)
+++ mesos/site/source/documentation/latest/powered-by-mesos.md Mon Mar 9 17:01:57 2015
@@ -27,6 +27,7 @@ layout: documentation
* [HubSpot](http://www.hubspot.com)
* [Ignidata](http://www.ignidata.com)
* [iQIYI](http://www.iqiyi.com/)
+* [Learnium](http://www.learnium.net/)
* [LIFX](http://lifx.co)
* [Localsensor](http://www.localsensor.com)
* [Magine TV](https://magine.com)
@@ -49,6 +50,7 @@ layout: documentation
* [Sigmoid Analytics](http://www.sigmoidanalytics.com/)
* [SiQueries](https://siqueries.com)
* [Squarespace](http://squarespace.com)
+* [TellApart](http://www.tellapart.com/)
* [The Factory](http://www.thefactory.com)
* Time Warner Cable
* [Twitter](http://www.twitter.com)
Modified: mesos/site/source/documentation/latest/release-guide.md
URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/release-guide.md?rev=1665304&r1=1665303&r2=1665304&view=diff
==============================================================================
--- mesos/site/source/documentation/latest/release-guide.md (original)
+++ mesos/site/source/documentation/latest/release-guide.md Mon Mar 9 17:01:57 2015
@@ -142,3 +142,10 @@ Per the guidelines [when to archive](htt
## Set the release date
1. Find the released Mesos version on https://issues.apache.org/jira/plugins/servlet/project-config/MESOS/versions, and update the release date.
+
+## Update external tooling
+
+1. Update the Mesos Homebrew package.
+ 1. Update the [Homebrew formula for Mesos](https://github.com/Homebrew/homebrew/blob/master/Library/Formula/mesos.rb) and test.
+ 1. Submit a PR to the [Homebrew repo](https://github.com/Homebrew/homebrew).
+ 1. Once accepted, verify that `brew install mesos` works.
Modified: mesos/site/source/documentation/latest/slave-recovery.md
URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/slave-recovery.md?rev=1665304&r1=1665303&r2=1665304&view=diff
==============================================================================
--- mesos/site/source/documentation/latest/slave-recovery.md (original)
+++ mesos/site/source/documentation/latest/slave-recovery.md Mon Mar 9 17:01:57 2015
@@ -27,25 +27,25 @@ As part of this feature, 4 new flags wer
* `checkpoint` : Whether to checkpoint slave and frameworks information
to disk [Default: true].
- This enables a restarted slave to recover status updates and reconnect
- with (--recover=reconnect) or kill (--recover=kill) old executors.
+ with (--recover=reconnect) or kill (--recover=cleanup) old executors.
> NOTE: From Mesos 0.22.0 this flag will be removed as it will be enabled for all slaves.
- * `strict` : Whether to do recovery in strict mode [Default: true].
- * If strict=true, any and all recovery errors are considered fatal.
- * If strict=false, any errors (e.g., corruption in checkpointed data) during recovery are
+* `strict` : Whether to do recovery in strict mode [Default: true].
+ - If strict=true, any and all recovery errors are considered fatal.
+ - If strict=false, any errors (e.g., corruption in checkpointed data) during recovery are
ignored and as much state as possible is recovered.
- * `recover` : Whether to recover status updates and reconnect with old executors [Default: reconnect].
- * If recover=reconnect, Reconnect with any old live executors.
- * If recover=cleanup, Kill any old live executors and exit.
+* `recover` : Whether to recover status updates and reconnect with old executors [Default: reconnect].
+ - If recover=reconnect, Reconnect with any old live executors.
+ - If recover=cleanup, Kill any old live executors and exit.
Use this option when doing an incompatible slave or executor upgrade!).
- NOTE: If no checkpointing information exists, no recovery is performed
- and the slave registers with the master as a new slave.
+ > NOTE: If no checkpointing information exists, no recovery is performed
+ > and the slave registers with the master as a new slave.
- * `recovery_timeout` : Amount of time allotted for the slave to recover [Default: 15 mins].
- * If the slave takes longer than `recovery_timeout` to recover, any executors that are waiting to
- reconnect to the slave will self-terminate.
- NOTE: This flag is only applicable when `--checkpoint` is enabled.
+* `recovery_timeout` : Amount of time allotted for the slave to recover [Default: 15 mins].
+ - If the slave takes longer than `recovery_timeout` to recover, any executors that are waiting to
+ reconnect to the slave will self-terminate.
+ > NOTE: This flag is only applicable when `--checkpoint` is enabled.
> NOTE: If none of the frameworks have enabled checkpointing,
> executors/tasks of frameworks die when the slave dies and are not recovered.
@@ -65,4 +65,4 @@ As part of this feature, `FrameworkInfo`
## Upgrading to 0.14.0
-If you want to upgrade a running Mesos cluster to 0.14.0 to take advantage of slave recovery please follow the [upgrade instructions](/documentation/latest/upgrades/).
\ No newline at end of file
+If you want to upgrade a running Mesos cluster to 0.14.0 to take advantage of slave recovery please follow the [upgrade instructions](/documentation/latest/upgrades/).
Modified: mesos/site/source/documentation/latest/upgrades.md
URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/upgrades.md?rev=1665304&r1=1665303&r2=1665304&view=diff
==============================================================================
--- mesos/site/source/documentation/latest/upgrades.md (original)
+++ mesos/site/source/documentation/latest/upgrades.md Mon Mar 9 17:01:57 2015
@@ -6,7 +6,14 @@ layout: documentation
This document serves as a guide for users who wish to upgrade an existing mesos cluster. Some versions require particular upgrade techniques when upgrading a running cluster. Some upgrades will have incompatible changes.
-## (WIP) Upgrading from 0.21.x to 0.22.x
+## Upgrading from 0.21.x to 0.22.x
+
+**NOTE** Slave checkpoint flag has been removed as it will be enabled for all
+slaves. Frameworks must still enable checkpointing during registration to take advantage
+of checkpointing their tasks.
+
+**NOTE** The stats.json endpoints for masters and slaves have been deprecated.
+Please refer to the metrics/snapshot endpoint.
**NOTE** The C++/Java/Python scheduler bindings have been updated. In particular, the driver can be constructed with an additional argument that specifies whether to use implicit driver acknowledgements. In `statusUpdate`, the `TaskStatus` now includes a UUID to make explicit acknowledgements possible.