You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mesos.apache.org by me...@apache.org on 2015/07/25 02:39:24 UTC
svn commit: r1692599 [5/5] - in /mesos/site: publish/ publish/blog/
publish/blog/framework-authentication-in-apache-mesos-0-15-0/
publish/blog/mesos-0-14-2-released/ publish/blog/mesos-0-16-0-released/
publish/blog/mesos-0-17-0-released-featuring-autor...
Modified: mesos/site/source/documentation/latest/app-framework-development-guide.md
URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/app-framework-development-guide.md?rev=1692599&r1=1692598&r2=1692599&view=diff
==============================================================================
--- mesos/site/source/documentation/latest/app-framework-development-guide.md (original)
+++ mesos/site/source/documentation/latest/app-framework-development-guide.md Sat Jul 25 00:39:22 2015
@@ -246,6 +246,6 @@ The `name` field is a string that that p
The `environment`, `location`, and `version` fields provide first class support for common attributes used to differentiate between similar services in large deployments. The `environment` may receive values such as `PROD/QA/DEV`, the `location` field may receive values like `EAST-US/WEST-US/EUROPE/AMEA`, and the `version` field may receive values like v2.0/v0.9. The exact use of these fields is up to the service discovery system.
-The `ports` field allows the framework to identify the ports a task listens to and explicitly name the functionality they represent and the layer-4 protocol they use (TCP, UDP, or other). For example, a Cassandra task will define ports like `â7000,Cluster,TCPâ`, `â7001,SSL,TCPâ`, `â9160,Thrift,TCPâ`, `â9042,Native,TCPâ`, and `â7199,JMX,TCPâ`. It is up to the service discovery system to use these names and protocol in appropriate ways, potentially combining them with the `name` field in DiscoveryInfo.
+The `ports` field allows the framework to identify the ports a task listens to and explicitly name the functionality they represent and the layer-4 protocol they use (TCP, UDP, or other). For example, a Cassandra task will define ports like `"7000,Cluster,TCP"`, `"7001,SSL,TCP"`, `"9160,Thrift,TCP"`, `"9042,Native,TCP"`, and `"7199,JMX,TCP"`. It is up to the service discovery system to use these names and protocol in appropriate ways, potentially combining them with the `name` field in DiscoveryInfo.
The `labels` field allows a framework to pass arbitrary labels to the service discovery system in the form of key/value pairs. Note that anything passed through this field is not guaranteed to be supported moving forward. Nevertheless, this field provides extensibility. Common uses of this field will allow us to identify use cases that require first class support.
Modified: mesos/site/source/documentation/latest/configuration.md
URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/configuration.md?rev=1692599&r1=1692598&r2=1692599&view=diff
==============================================================================
--- mesos/site/source/documentation/latest/configuration.md (original)
+++ mesos/site/source/documentation/latest/configuration.md Sat Jul 25 00:39:22 2015
@@ -87,12 +87,21 @@ If you have special compilation requirem
--ip=VALUE
</td>
<td>
- IP address to listen on
-
+ IP address to listen on; this cannot be used in conjunction
+ with --ip_discovery_command.
</td>
</tr>
<tr>
<td>
+ --ip_discovery_command=VALUE
+ </td>
+ <td>
+ Optional IP discovery command: if set, it is expected to emit
+ the IP address which Master will try to bind to. Cannot be used
+ in conjunction with --ip.
+ </td>
+ </tr> <tr>
+ <td>
--log_dir=VALUE
</td>
<td>
@@ -970,21 +979,21 @@ file:///path/to/file (where file contain
</tr>
<tr>
<td>
- --docker_sandbox_directory=VALUE
+ --docker_stop_timeout=VALUE
</td>
<td>
- The absolute path for the directory in the container where the
- sandbox is mapped to.
- (default: /mnt/mesos/sandbox)
+ The time as a duration for docker to wait after stopping an instance
+ before it kills that instance. (default: 0secs)
</td>
</tr>
<tr>
<td>
- --docker_stop_timeout=VALUE
+ --sandbox_directory=VALUE
</td>
<td>
- The time as a duration for docker to wait after stopping an instance
- before it kills that instance. (default: 0secs)
+ The absolute path for the directory in the container where the
+ sandbox is mapped to.
+ (default: /mnt/mesos/sandbox)
</td>
</tr>
<tr>
Modified: mesos/site/source/documentation/latest/fetcher-cache-internals.md
URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/fetcher-cache-internals.md?rev=1692599&r1=1692598&r2=1692599&view=diff
==============================================================================
--- mesos/site/source/documentation/latest/fetcher-cache-internals.md (original)
+++ mesos/site/source/documentation/latest/fetcher-cache-internals.md Sat Jul 25 00:39:22 2015
@@ -87,7 +87,7 @@ The implementation is oriented at this c
- b) Wait for size fetching combined and then space reservation for new cache entries.
2. After making fetcher cache items and running mesos-fetcher.
-- Complete new cache items with success/failure, which as an important side-effect informs concurrent fetch runsâ futures in phase 1/a.
+- Complete new cache items with success/failure, which as an important side-effect informs concurrent fetch runs' futures in phase 1/a.
The futures for phase 1 are not shared outside one fetch run. They exclusively guard asynchronous operations for the same fetch run. Their type parameter does not really matter. But each needs to correspond to one URI and eventual fetch item somehow. Multiple variants have been proposed for this. The complexity remains about the same.
Modified: mesos/site/source/documentation/latest/operational-guide.md
URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/operational-guide.md?rev=1692599&r1=1692598&r2=1692599&view=diff
==============================================================================
--- mesos/site/source/documentation/latest/operational-guide.md (original)
+++ mesos/site/source/documentation/latest/operational-guide.md Sat Jul 25 00:39:22 2015
@@ -23,7 +23,7 @@ For online reconfiguration of the log, s
### Increasing the quorum size
As the size of a cluster grows, it may be desired to increase the quorum size for additional fault tolerance.
-The following steps indicate how to increment the quorum size, using 3 â 5 masters as an example (quorum size 2 â 3):
+The following steps indicate how to increment the quorum size, using 3 -> 5 masters as an example (quorum size 2 -> 3):
1. Initially, 3 masters are running with `--quorum=2`
2. Restart the original 3 masters with `--quorum=3`
@@ -34,7 +34,7 @@ To increase the quorum by N, repeat this
### Decreasing the quorum size
-The following steps indicate how to decrement the quorum size, using 5 â 3 masters as an example (quorum size 3 â 2):
+The following steps indicate how to decrement the quorum size, using 5 -> 3 masters as an example (quorum size 3 -> 2):
1. Initially, 5 masters are running with `--quorum=3`
2. Remove 2 masters from the cluster, ensure they will not be restarted (See NOTE section above). Now 3 masters are running with `--quorum=3`
@@ -44,4 +44,4 @@ To decrease the quorum by N, repeat this
### Replacing a master
-Please see the NOTE section above. So long as the failed master is guaranteed to not re-join the ensemble, it is safe to start a new master _with an empty log_ and allow it to catch up.
\ No newline at end of file
+Please see the NOTE section above. So long as the failed master is guaranteed to not re-join the ensemble, it is safe to start a new master _with an empty log_ and allow it to catch up.
Added: mesos/site/source/documentation/latest/oversubscription.md
URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/oversubscription.md?rev=1692599&view=auto
==============================================================================
--- mesos/site/source/documentation/latest/oversubscription.md (added)
+++ mesos/site/source/documentation/latest/oversubscription.md Sat Jul 25 00:39:22 2015
@@ -0,0 +1,305 @@
+--- layout: documentation ---
+
+# Oversubscription
+
+High-priority user-facing services are typically provisioned on large clusters
+for peak load and unexpected load spikes. Hence, for most of time, the
+provisioned resources remain underutilized. Oversubscription takes advantage of
+temporarily unused resources to execute best-effort tasks such as background
+analytics, video/image processing, chip simulations, and other low priority
+jobs.
+
+## How does it work?
+
+Oversubscription was introduced in Mesos 0.23.0 and adds two new slave
+components: a Resource Estimator and a Quality of Service (QoS) Controller,
+alongside extending the existing resource allocator, resource monitor, and
+mesos slave. The new components and their interactions are illustrated below.
+
+
+
+### Resource estimation
+
+ - (1) The first step is to identify the amount of oversubscribed resources.
+ The resource estimator taps into the resource monitor and periodically gets
+usage statistics via `ResourceStatistic` messages. The resource estimator
+applies logic based on the collected resource statistics to determine the
+amount of oversubscribed resources. This can be a series of control algorithms
+based on measured resource usage slack (allocated but unused resources) and
+allocation slack.
+
+ - (2) The slave keeps polling estimates from the resource estimator and tracks
+ the latest estimate.
+
+ - (3) The slave will send the total amount of oversubscribed resources to the
+ master when the latest estimate is different from the previous estimate.
+
+### Resource tracking & scheduling algorithm
+
+ - (4) The allocator keeps track of the oversubscribed resources separately
+ from regular resources and annotate those resources as `revocable`. It is up
+to the resource estimator to determine which types of resources can be
+oversubscribed. It is recommended only to oversubscribe _compressible_
+resources such as cpu shares, bandwidth, etc.
+
+### Frameworks
+
+ - (5) Frameworks can choose to launch tasks on revocable resources by using
+ the regular launchTasks() API. To safe-guard frameworks that are not
+designed to deal with preemption, only frameworks registering with the
+`REVOCABLE_RESOURCES` capability set in its framework info will receive offers
+with revocable resources. Further more, recovable resources cannot be
+dynamically reserved and persistent volumes should not be created on revocable
+disk resources.
+
+### Task launch
+
+ - The revocable task is launched as usual when the runTask request is received
+ on the slave. The resources will still be marked as revocable and isolators
+can take appropriate actions, if certain resources need to be setup differently
+for revocable and regular tasks.
+
+> NOTE: If any resource used by a task or executor is
+revocable, the whole container is treated as a revocable container and can
+therefore be killed or throttled by the QoS Controller.
+
+### Interference detection
+
+ - (6) When the revocable task is running, it is important to constantly
+ monitor the original task running on those resources and guarantee
+performance based on an SLA. In order to react to detected interference, the
+QoS controller needs to be able to kill or throttle running revocable tasks.
+
+## Enabling frameworks to use oversubscribed resources
+
+Frameworks planning to use oversubscribed resources need to register with the
+`REVOCABLE_RESOURCES` capability set:
+
+~~~{.cpp}
+FrameworkInfo framework;
+framework.set_name("Revocable framework");
+
+framework.add_capabilities()->set_type(
+ FrameworkInfo::Capability::REVOCABLE_RESOURCES);
+~~~
+
+From that point on, the framework will start to receive revocable resources in
+offers.
+
+> NOTE: That there is no guarantee that the Mesos cluster has oversubscription
+enabled. If not, no revocable resources will be offered. See below for
+instructions how to configure Mesos for oversubscription.
+
+### Launching tasks using revocable resources
+
+Launching tasks using recovable resources is done through the existing
+`launchTasks` API. Revocable resources will have the `recovable` field set. See
+below for an example offer with regular and revocable resources.
+
+~~~{.json}
+{
+ "id": "20150618-112946-201330860-5050-2210-0000",
+ "framework_id": "20141119-101031-201330860-5050-3757-0000",
+ "slave_id": "20150618-112946-201330860-5050-2210-S1",
+ "hostname": "foobar",
+ "resources": [
+ {
+ "name": "cpus",
+ "type": "SCALAR",
+ "scalar": {
+ "value": 2.0
+ },
+ "role": "*"
+ }, {
+ "name": "mem",
+ "type": "SCALAR",
+ "scalar": {
+ "value": 512.0
+ },
+ "role": "*"
+ },
+ {
+ "name": "cpus",
+ "type": "SCALAR",
+ "scalar": {
+ "value": 0.45
+ },
+ "role": "*",
+ "revocable": {}
+ }
+ ]
+}
+~~~
+
+## Writing a custom resource estimator
+
+The resource estimator estimates and predicts the total resources used on the
+slave and informs the master about resources that can be oversubscribed. By
+default, Mesos comes with a `noop` and a `fixed` resource estimator. The `noop`
+estimator only provides an empty estimate to the slave and stalls, effectively
+disabling oversubscription. The `fixed` estimator doesn't use the actual
+measured slack, but oversubscribes the node with fixed resource amount (defined
+via a command line flag).
+
+The interface is defined below:
+
+~~~{.cpp}
+class ResourceEstimator
+{
+public:
+ // Initializes this resource estimator. This method needs to be
+ // called before any other member method is called. It registers
+ // a callback in the resource estimator. The callback allows the
+ // resource estimator to fetch the current resource usage for each
+ // executor on slave.
+ virtual Try<Nothing> initialize(
+ const lambda::function<process::Future<ResourceUsage>()>& usage) = 0;
+
+ // Returns the current estimation about the *maximum* amount of
+ // resources that can be oversubscribed on the slave. A new
+ // estimation will invalidate all the previously returned
+ // estimations. The slave will be calling this method periodically
+ // to forward it to the master. As a result, the estimator should
+ // respond with an estimate every time this method is called.
+ virtual process::Future<Resources> oversubscribable() = 0;
+};
+~~~
+
+## Writing a custom QoS controller
+
+The interface for implementing custom QoS Controllers is defined below:
+
+~~~{.cpp}
+class QoSController
+{
+public:
+ // Initializes this QoS Controller. This method needs to be
+ // called before any other member method is called. It registers
+ // a callback in the QoS Controller. The callback allows the
+ // QoS Controller to fetch the current resource usage for each
+ // executor on slave.
+ virtual Try<Nothing> initialize(
+ const lambda::function<process::Future<ResourceUsage>()>& usage) = 0;
+
+ // A QoS Controller informs the slave about corrections to carry
+ // out, but returning futures to QoSCorrection objects. For more
+ // information, please refer to mesos.proto.
+ virtual process::Future<std::list<QoSCorrection>> corrections() = 0;
+};
+~~~
+
+> NOTE The QoS Controller must not block `corrections()`. Back the QoS
+> Controller with it's own libprocess actor instead.
+
+The QoS Controller informs the slave that particular corrective actions need to
+be made. Each corrective action contains information about executor or task and
+the type of action to perform.
+
+~~~{.proto}
+message QoSCorrection {
+ enum Type {
+ KILL = 1; // Terminate an executor.
+ }
+
+ message Kill {
+ optional FrameworkID framework_id = 1;
+ optional ExecutorID executor_id = 2;
+ }
+
+ required Type type = 1;
+ optional Kill kill = 2;
+}
+~~~
+
+## Configuring Mesos for oversubscription
+
+Five new flags has been added to the slave:
+
+<table class="table table-striped">
+ <thead>
+ <tr>
+ <th width="30%">
+ Flag
+ </th>
+ <th>
+ Explanation
+ </th>
+ </thead>
+
+ <tr>
+ <td>
+ --oversubscribed_resources_interval=VALUE
+ </td>
+ <td>
+ The slave periodically updates the master with the current estimation
+about the total amount of oversubscribed resources that are allocated and
+available. The interval between updates is controlled by this flag. (default:
+15secs)
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ --qos_controller=VALUE
+ </td>
+ <td>
+ The name of the QoS Controller to use for oversubscription.
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ --qos_correction_interval_min=VALUE
+ </td>
+ <td>
+ The slave polls and carries out QoS corrections from the QoS Controller
+based on its observed performance of running tasks. The smallest interval
+between these corrections is controlled by this flag. (default: 0ns)
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ --resource_estimator=VALUE
+ </td>
+ <td>
+ The name of the resource estimator to use for oversubscription.
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ --resource_monitoring_interval=VALUE
+ </td>
+ <td>
+ Periodic time interval for monitoring executor resource usage (e.g.,
+10secs, 1min, etc) (default: 1secs)
+ </td>
+ </tr>
+
+</table>
+
+The `fixed` resource estimator is enabled as follows:
+
+```
+--resource_estimator="org_apache_mesos_FixedResourceEstimator"
+
+--modules='{
+ "libraries": {
+ "file": "/usr/local/lib64/libfixed_resource_estimator.so",
+ "modules": {
+ "name": "org_apache_mesos_FixedResourceEstimator",
+ "parameters": {
+ "key": "resources",
+ "value": "cpus:14"
+ }
+ }
+ }
+}'
+```
+
+In the example above, a fixed amount of 14 cpus will be offered as revocable
+resources.
+
+To select custom a resource estimator and QoS controller, please refer to the
+[modules documentation](/documentation/latest/modules/).
Added: mesos/site/source/documentation/latest/persistent-volume.md
URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/persistent-volume.md?rev=1692599&view=auto
==============================================================================
--- mesos/site/source/documentation/latest/persistent-volume.md (added)
+++ mesos/site/source/documentation/latest/persistent-volume.md Sat Jul 25 00:39:22 2015
@@ -0,0 +1,231 @@
+---
+layout: documentation
+---
+
+# Persistent Volume
+
+Mesos provides a mechanism to create a persistent volume from disk resources.
+This enables stateful services such as HDFS and Cassandra to store their data
+within Mesos rather than having to resort to network-mounted EBS volumes that
+needs to be placed in a well-known location.
+
+Persistent volumes can only be created from __reserved__ disk resources, whether
+it be statically reserved or dynamically reserved. A dynamically reserved
+persistent volume also cannot be unreserved without having explicitly destroyed
+the volume. These rules exist to limit the accidental mistakes such as:
+a persistent volume containing sensitive data being offered to other frameworks
+in the cluster.
+
+Please refer to the
+[Reservation](/documentation/latest/reservation/) documentation for details regarding reservation
+mechanisms available in Mesos.
+
+Persistent volumes can be created by __operators__ and authorized
+__frameworks__. We require a `principal` from the operator or framework in order
+to authenticate/authorize the operations. [Authorization](/documentation/latest/authorization/) is
+specified via the existing ACL mechanism. (___Coming Soon___)
+
+* `Offer::Operation::Create` and `Offer::Operation::Destroy` messages are
+ available for __frameworks__ to send back via the `acceptOffers` API as a
+ response to a resource offer.
+* `/create` and `/destroy` HTTP endpoints are available for __operators__
+ to manage persistent volumes through the master. (___Coming Soon___).
+
+In the following sections, we will walk through examples of each of the
+interfaces described above.
+
+
+## `Offer::Operation::Create`
+
+A framework is able to create volumes through the resource offer cycle.
+Suppose we receive a resource offer with 2048 MB of dynamically reserved disk.
+
+```
+{
+ "id" : <offer_id>,
+ "framework_id" : <framework_id>,
+ "slave_id" : <slave_id>,
+ "hostname" : <hostname>,
+ "resources" : [
+ {
+ "name" : "disk",
+ "type" : "SCALAR",
+ "scalar" : { "value" : 2048 },
+ "role" : <framework_role>,
+ "reservation" : {
+ "principal" : <framework_principal>
+ }
+ }
+ ]
+}
+```
+
+We can create a persistent volume from the 2048 MB of disk resources by sending
+the following `Offer::Operation` message via the `acceptOffers` API.
+`Offer::Operation::Create` has a `volumes` field which we specify with the
+persistent volume information. We need to specify the following:
+
+1. ID of the persistent volume which needs to be unique per role on each slave.
+1. The non-nested relative path within the container to mount the volume.
+1. The permissions for the volume. Currently, `"RW"` is the only possible value.
+
+```
+{
+ "type" : Offer::Operation::CREATE,
+ "create": {
+ "volumes" : [
+ {
+ "name" : "disk",
+ "type" : "SCALAR",
+ "scalar" : { "value" : 2048 },
+ "role" : <framework_role>,
+ "reservation" : {
+ "principal" : <framework_principal>
+ },
+ "disk": {
+ "persistence": {
+ "id" : <persistent_volume_id>
+ },
+ "volume" : {
+ "container_path" : <container_path>,
+ "mode" : <mode>
+ }
+ }
+ }
+ ]
+ }
+}
+```
+
+The subsequent resource offer will __contain__ the following persistent volume:
+
+```
+{
+ "id" : <offer_id>,
+ "framework_id" : <framework_id>,
+ "slave_id" : <slave_id>,
+ "hostname" : <hostname>,
+ "resources" : [
+ {
+ "name" : "disk",
+ "type" : "SCALAR",
+ "scalar" : { "value" : 2048 },
+ "role" : <framework_role>,
+ "reservation" : {
+ "principal" : <framework_principal>
+ },
+ "disk": {
+ "persistence": {
+ "id" : <persistent_volume_id>
+ },
+ "volume" : {
+ "container_path" : <container_path>,
+ "mode" : <mode>
+ }
+ }
+ }
+ ]
+}
+```
+
+
+## `Offer::Operation::Destroy`
+
+A framework is able to destroy persistent volumes through the resource offer
+cycle. In [Offer::Operation::Create](#offeroperationcreate), we created a
+persistent volume from 2048 MB of disk resources. Mesos will not garbage-collect
+this volume until we explicitly destroy it. Suppose we would like to destroy the
+volume we created. First, we receive a resource offer (copy/pasted from above):
+
+```
+{
+ "id" : <offer_id>,
+ "framework_id" : <framework_id>,
+ "slave_id" : <slave_id>,
+ "hostname" : <hostname>,
+ "resources" : [
+ {
+ "name" : "disk",
+ "type" : "SCALAR",
+ "scalar" : { "value" : 2048 },
+ "role" : <framework_role>,
+ "reservation" : {
+ "principal" : <framework_principal>
+ },
+ "disk": {
+ "persistence": {
+ "id" : <persistent_volume_id>
+ },
+ "volume" : {
+ "container_path" : <container_path>,
+ "mode" : <mode>
+ }
+ }
+ }
+ ]
+}
+```
+
+We destroy the persistent volume by sending the `Offer::Operation` message via
+the `acceptOffers` API. `Offer::Operation::Destroy` has a `volumes` field which
+we specify the persistent volumes to be destroyed.
+
+```
+{
+ "type" : Offer::Operation::DESTROY,
+ "destroy" : {
+ "volumes" : [
+ {
+ "name" : "disk",
+ "type" : "SCALAR",
+ "scalar" : { "value" : 2048 },
+ "role" : <framework_role>,
+ "reservation" : {
+ "principal" : <framework_principal>
+ },
+ "disk": {
+ "persistence": {
+ "id" : <persistent_volume_id>
+ },
+ "volume" : {
+ "container_path" : <container_path>,
+ "mode" : <mode>
+ }
+ }
+ }
+ ]
+ }
+}
+```
+
+The persistent volume will be destroyed, but the disk resources will still be
+reserved. As such, the subsequent resource offer will __contain__ the following
+reserved disk resources:
+
+```
+{
+ "id" : <offer_id>,
+ "framework_id" : <framework_id>,
+ "slave_id" : <slave_id>,
+ "hostname" : <hostname>,
+ "resources" : [
+ {
+ "name" : "disk",
+ "type" : "SCALAR",
+ "scalar" : { "value" : 2048 },
+ "role" : <framework_role>,
+ "reservation" : {
+ "principal" : <framework_principal>
+ }
+ }
+ ]
+}
+```
+
+Note that in 0.23, even after you destroy the persistent volume, its content
+will still be on the disk. The garbage collection for persistent volumes is
+coming soon: [MESOS-2048](https://issues.apache.org/jira/browse/MESOS-2408).
+
+
+### `/create` (_Coming Soon_)
+### `/destroy` (_Coming Soon_)
Modified: mesos/site/source/documentation/latest/reconciliation.md
URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/reconciliation.md?rev=1692599&r1=1692598&r2=1692599&view=diff
==============================================================================
--- mesos/site/source/documentation/latest/reconciliation.md (original)
+++ mesos/site/source/documentation/latest/reconciliation.md Sat Jul 25 00:39:22 2015
@@ -77,10 +77,10 @@ The technique for performing reconciliat
tasks, until an update is received for each task, using exponential backoff:
1. let `start = now()`
-2. let `remaining = { T ϵ tasks | T is non-terminal }`
+2. let `remaining = { T in tasks | T is non-terminal }`
3. Perform reconciliation: `reconcile(remaining)`
4. Wait for status updates to arrive (use truncated exponential backoff). For each update, note the time of arrival.
-5. let `remaining = { T ϵ remaining | T.last_update_arrival() < start }`
+5. let `remaining = { T in remaining | T.last_update_arrival() < start }`
6. If `remaining` is non-empty, go to 3.
This reconciliation algorithm **must** be run after each (re-)registration.
Modified: mesos/site/source/documentation/latest/release-guide.md
URL: http://svn.apache.org/viewvc/mesos/site/source/documentation/latest/release-guide.md?rev=1692599&r1=1692598&r2=1692599&view=diff
==============================================================================
--- mesos/site/source/documentation/latest/release-guide.md (original)
+++ mesos/site/source/documentation/latest/release-guide.md Sat Jul 25 00:39:22 2015
@@ -13,10 +13,16 @@ This guide describes the process of doin
1. Ensure that you have a GPG key or generate a new one, e.g., using `gpg --gen-key`.
2. Add your GPG public key to the Apache Mesos dist repository in the KEYS file.
- - Fetch the svn repository `svn co https://dist.apache.org/repos/dist/release/mesos`
- - Append your public key using one of methods described in KEYS,
- e.g., `(gpg --list-sigs <your name> && gpg --armor --export <your name>) >> KEYS`.
- - Push the commit: `svn ci`
+
+ - Fetch the svn repository:<br>
+ `svn co https://dist.apache.org/repos/dist/release/mesos`
+
+ - Append your public key using one of methods described in KEYS,
+ e.g.,<br>
+ `(gpg --list-sigs <your name> && gpg --armor --export <your name>) >> KEYS`.
+
+ - Push the commit:<br>
+ `svn ci`
3. Submit your GPG public key to a keyserver, e.g., [MIT PGP Public Key Server](https://pgp.mit.edu).
@@ -51,17 +57,26 @@ This guide describes the process of doin
1. Go to [Apache Jira](https://issues.apache.org/jira/browse/MESOS) and make sure that
the CHANGELOG for the release version is up to date.
- > NOTE: You should move all **Unresolved** tickets marked with `Fix Version` as the
- release version to the next release version.
+ > NOTE: You should move all **Unresolved** tickets marked with `Fix Version`
+ or `Target Version` as the release version to the next release version.
+
+ > PROTIP: Use a JIRA dashboard [(example)](https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12326227)
+ to track the progress of targeted issues as the release date approaches.
> PROTIP: Use `bulk edit` option in JIRA to move the tickets and make sure to
- **uncheck** the option that emails everyone about the move.
+ **uncheck** the option that emails everyone about the move.
-2. Update and commit the `CHANGELOG` for the release.
+2. Update and commit the `CHANGELOG` for the release. For major releases we like to call
+ out any major features, API changes, or deprecations.
> NOTE: You should use JIRA to generate the CHANGELOG for you. Click on the release
- version in [JIRA](https://issues.apache.org/jira/browse/MESOS#selectedTab=com.atlassian.jira.plugin.system.project%3Aversions-panel) and click
- on the `Release Notes`. Make sure to configure the release notes in text format.
+ version in [JIRA](https://issues.apache.org/jira/browse/MESOS#selectedTab=com.atlassian.jira.plugin.system.project%3Aversions-panel) and click
+ on the `Release Notes`. Make sure to configure the release notes in text format.
+
+ > NOTE: The JIRA Release Notes will list only tickets with `Fix Version` set
+ to that version. You should check for any Resolved tickets that have
+ `Target Version` set but not `Fix Version`. Also check for any Unresolved
+ or `Duplicate`/`Invalid` tickets that incorrectly set the `Fix Version`.
3. If not already done, update and commit `configure.ac` for the release.
@@ -71,14 +86,15 @@ This guide describes the process of doin
5. Update and commit `docs/upgrades.md` with instructions about how to upgrade
a live cluster from the previous release version to this release version.
-6. If this is a major release please write and commit documentation for this feature.
+6. If this is a major release, please ensure that user documentation has been
+ added for any new features.
## Tagging the release candidate
1. Ensure that you can build and pass all the tests.
- $ make -j3 distcheck
+ $ sudo make -j3 distcheck
2. First tag the required SHA locally.
@@ -95,6 +111,9 @@ This guide describes the process of doin
> NOTE: This script assumes that you have the requisite permissions to deploy the JAR. For
instructions on how to set it up, please refer to `src/java/MESOS-MAVEN-README`.
+ > NOTE: gnu-sed (Linux) requires `-i''` instead of the `-i ''` (space-separated) that default OSX uses.
+ You may need to modify your local copy of tag.sh for it to complete successfully.
+
4. It is not uncommon to release multiple release candidates, with increasing release candidate
version, if there are bugs found.
@@ -111,6 +130,24 @@ This guide describes the process of doin
3. The release script also spits out an email template that you could use to send the vote email.
+ > NOTE: The `date -v+3d` command does not work on some platforms (e.g. Ubuntu),
+ so you may need to fill in the vote end date manually. The vote should last
+ for 3 business days instead of 3 calendar days anyway. Sometimes we prefer a
+ weeklong vote, to allow more time for integration testing.
+
+## Preparing a new release candidate
+
+1. If the vote does not pass (any -1s or showstopper bugs), track the issues as new JIRAs for the release.
+
+2. When all known issues are resolved, update the CHANGELOG with the newly fixed JIRAs.
+
+3. Once all patches are committed to master, cherry-pick them on top of the previous release candidate tag.
+ This is the same process used for point releases (e.g. 0.22.1) as well.
+
+ $ git checkout X.Y.Z-rcR
+ $ git cherry-pick abcdefgh...
+
+4. Now go back up to the "Tagging the release candidate" section and repeat.
## Releasing the release candidate
@@ -129,9 +166,14 @@ This guide describes the process of doin
## Updating the website
-1. After a successful release please update the website pointing to the new release.
+1. After a successful release, please update the website pointing to the new release.
+ See our [website README](http://svn.apache.org/repos/asf/mesos/site/) and
+ the general [Apache project website guide](https://www.apache.org/dev/project-site.html)
+ for details on how to build and publish the website.
-2. It is also recommended to write a blog post announcing the feature.
+ $ svn co https://svn.apache.org/repos/asf/mesos/site mesos-site
+
+2. Write a blog post announcing the new release and its features and major bug fixes.
## Remove old releases from svn
@@ -143,11 +185,21 @@ Per the guidelines [when to archive](htt
## Set the release date
-1. Find the released Mesos version on https://issues.apache.org/jira/plugins/servlet/project-config/MESOS/versions, and update the release date.
+1. Find the released Mesos version on https://issues.apache.org/jira/plugins/servlet/project-config/MESOS/versions, and "release" it with the correct release date.
## Update external tooling
-1. Update the Mesos Homebrew package.
+Upload the mesos.interface package to PyPi.
+
+ 1. Create/use a PyPi account with access to the [mesos.interface submit form](https://pypi.python.org/pypi?name=mesos.interface&:action=submit_form).
+ You may need to ask a current package owner to add you as an owner/maintainer.
+ 1. Setup your [`~/.pypirc`](https://docs.python.org/2/distutils/packageindex.html#pypirc) with your PyPi username and password.
+ 1. After a successful Mesos `make` (any architecture), cd to `build/src/python/interface`.
+ 1. Run `python setup.py register` to register this package.
+ 1. Run `python setup.py bdist_egg upload` to upload the egg for this package.
+
+Update the Mesos Homebrew package.
+
1. Update the [Homebrew formula for Mesos](https://github.com/Homebrew/homebrew/blob/master/Library/Formula/mesos.rb) and test.
1. Submit a PR to the [Homebrew repo](https://github.com/Homebrew/homebrew).
1. Once accepted, verify that `brew install mesos` works.
Modified: mesos/site/source/downloads.html.md
URL: http://svn.apache.org/viewvc/mesos/site/source/downloads.html.md?rev=1692599&r1=1692598&r2=1692599&view=diff
==============================================================================
--- mesos/site/source/downloads.html.md (original)
+++ mesos/site/source/downloads.html.md Sat Jul 25 00:39:22 2015
@@ -8,8 +8,10 @@ breadcrumb: Downloads
<h4>Previous Releases</h4>
<ul>
+ <li><a href="http://archive.apache.org/dist/mesos/0.23.0/">0.23.0</a>
+ (<a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311242&version=12328728">Release Notes</a>)</li>
<li><a href="http://archive.apache.org/dist/mesos/0.22.1/">0.22.1</a>
- (<a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311242&version=12329855">Release Notes</a>)</li>
+ (<a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311242&version=12329855">Release Notes</a>)</li>
<li><a href="http://archive.apache.org/dist/mesos/0.22.0/">0.22.0</a>
(<a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311242&version=12328650">Release Notes</a>)</li>
<li><a href="http://archive.apache.org/dist/mesos/0.21.1/">0.21.1</a>
@@ -58,8 +60,8 @@ breadcrumb: Downloads
<div class="col-md-8">
<h1>Download Mesos</h1>
<p>Download the most recent stable release:
- <a href="http://www.apache.org/dyn/mirrors/mirrors.cgi/mesos/0.22.1/">0.22.1</a>
- (<a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311242&version=12329855">Release Notes</a>)
+ <a href="http://www.apache.org/dyn/mirrors/mirrors.cgi/mesos/0.23.0/">0.23.0</a>
+ (<a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311242&version=12328728">Release Notes</a>)
</p>
<h4>Getting the code via source control</h4>
Modified: mesos/site/source/index.html.md
URL: http://svn.apache.org/viewvc/mesos/site/source/index.html.md?rev=1692599&r1=1692598&r2=1692599&view=diff
==============================================================================
--- mesos/site/source/index.html.md (original)
+++ mesos/site/source/index.html.md Sat Jul 25 00:39:22 2015
@@ -7,9 +7,12 @@ title: Apache Mesos
<div class="col-md-7 nopadding">
<h1>Program against your datacenter like itâs a single pool of resources</h1>
<p class="lead">Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively.</p>
+
+ <a class="btn btn-lg btn-success" href="downloads/"><span class="glyphicon glyphicon-download"></span> Download Mesos 0.23.0</a>
+ <p>or learn how to <a href="gettingstarted/">get started</a></p>
</div>
<div class="col-md-5 text-center download">
- <a href="http://events.linuxfoundation.org/events/mesoscon/attend/register"><img src="/assets/img/mesoscon-2015-banner-small.png" alt="MesosCon Registration Banner" /></a>
+ <a href="http://events.linuxfoundation.org/events/mesoscon/attend/register"><img src="/assets/img/mesoscon-2015-banner-small.png" alt="MesosCon Registration Banner" /></a>
</div>
</div>
</div>
@@ -37,13 +40,11 @@ title: Apache Mesos
<div class="col-md-5">
<h3>News</h3>
<ul>
- <li><em>July 13, 2015</em> - MesosCon pre-conference and social activities announced! See the <a href="/blog/mesoscon-seattle-preconference-and-social-activities/">blog post announcement</a> for more details.</li>
- <li><em>July 1, 2015</em> - MesosCon Seattle 2015 keynote speakers announced! See the <a href="/blog/mesoscon-seattle-keynote-speakers-announced/">blog post announcement</a> for more details.</li>
- <li><em>May 12, 2015</em> - Mesos 0.22.1 is released! See the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311242&version=12329855">0.22.1 release notes</a> and <a href="/blog/mesos-0-22-1-released/">blog post announcement</a> for more details.</li>
- <li><em>March 25, 2015</em> - Mesos 0.22.0 is released! See the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311242&version=12328650">0.22.0 release notes</a> and <a href="/blog/mesos-0-22-0-released/">blog post announcement</a> for more details.</li>
- <li><em>January 20, 2015</em> - Mesos 0.21.1 is released! See the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12329076&projectId=12311242">0.21.1 release notes</a> and <a href="/blog/mesos-0-21-1-released/">blog post announcement</a> for more details.</li>
- <li><em>January 14, 2015</em> - <a href="/blog/mesoscon-2015-earlybird-registration-now-open/">MesosCon Early-Bird Registration Now Open</a> to the first 140 attendees that register for the conference. Join us in Seattle this August!</li>
- <li><em>November 17, 2014</em> - Mesos 0.21.0 is released! See the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311242&version=12327045">0.21.0 release notes</a> and <a href="/blog/mesos-0-21-0-released/">blog post announcement</a> for more details.</li>
+ <li><em>July 22, 2015</em> - Mesos 0.23.0 is released! See the <a href="https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.23.0">CHANGELOG</a> for more details. <!--a href="/blog/mesos-0-23-0-released/"-->Blog post<!--/a--> coming soon.</li>
+ <li><em>July 13, 2015</em> - MesosCon pre-conference and social activities announced! See the <a href="/blog/mesoscon-seattle-preconference-and-social-activities/">blog post</a> for more details.</li>
+ <li><em>July 1, 2015</em> - MesosCon Seattle 2015 keynote speakers announced! See the <a href="/blog/mesoscon-seattle-keynote-speakers-announced/">blog post</a> for more details.</li>
+ <li><em>May 12, 2015</em> - Mesos 0.22.1 is released! See the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311242&version=12329855">0.22.1 release notes</a> and <a href="/blog/mesos-0-22-1-released/">blog post</a> for more details.</li>
+ <li><em>March 25, 2015</em> - Mesos 0.22.0 is released! See the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311242&version=12328650">0.22.0 release notes</a> and <a href="/blog/mesos-0-22-0-released/">blog post</a> for more details.</li>
</ul>
<h3>Follow Us</h3>
Modified: mesos/site/source/layouts/layout.erb
URL: http://svn.apache.org/viewvc/mesos/site/source/layouts/layout.erb?rev=1692599&r1=1692598&r2=1692599&view=diff
==============================================================================
--- mesos/site/source/layouts/layout.erb (original)
+++ mesos/site/source/layouts/layout.erb Sat Jul 25 00:39:22 2015
@@ -82,7 +82,7 @@
<!-- footer -->
<div class="footer">
- <p>© 2012-2014 <a href="http://apache.org">The Apache Software Foundation</a>.
+ <p>© 2012-2015 <a href="http://apache.org">The Apache Software Foundation</a>.
Apache Mesos, the Apache feather logo, and the Apache Mesos project logo are trademarks of The Apache Software Foundation.<p>
</div><!-- /footer -->
@@ -92,4 +92,4 @@
<%= javascript_include_tag '//code.jquery.com/jquery-1.11.0.min.js' %>
<%= javascript_include_tag '//netdna.bootstrapcdn.com/bootstrap/3.1.1/js/bootstrap.min.js' %>
</body>
-</html>
\ No newline at end of file
+</html>