You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mesos.apache.org by vi...@apache.org on 2013/09/19 23:00:47 UTC
git commit: Updated CHANGELOG and docs for 0.14.0.
Updated Branches:
refs/heads/master 58eb26aaf -> 4cb7b74d1
Updated CHANGELOG and docs for 0.14.0.
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/4cb7b74d
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/4cb7b74d
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/4cb7b74d
Branch: refs/heads/master
Commit: 4cb7b74d10099f8f89a9871090fe2b5eb20ab62c
Parents: 58eb26a
Author: Vinod Kone <vi...@twitter.com>
Authored: Thu Sep 19 14:00:10 2013 -0700
Committer: Vinod Kone <vi...@twitter.com>
Committed: Thu Sep 19 14:00:10 2013 -0700
----------------------------------------------------------------------
CHANGELOG | 13 ++++++
docs/Configuration.md | 45 ++++++++++++++++++--
docs/Mesos-developers-guide.md | 42 +++++++++---------
docs/Running-Hadoop-on-Mesos.md | 33 --------------
...-Second-Instance-of-Hadoop-(Snow-Leopard).md | 26 -----------
5 files changed, 75 insertions(+), 84 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/mesos/blob/4cb7b74d/CHANGELOG
----------------------------------------------------------------------
diff --git a/CHANGELOG b/CHANGELOG
index d8e9c10..206cf28 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -21,6 +21,7 @@ Release Notes - Mesos - Version 0.14.0
* There are also several bug fixes and stability improvements.
+
** Sub-task
* [MESOS-548] - Upgrade angular.js to use the full angular-ui.js
* [MESOS-549] - Change truncated IDs to show on hover
@@ -37,6 +38,7 @@ Release Notes - Mesos - Version 0.14.0
* [MESOS-461] - Freezer failure while in FREEZING state.
* [MESOS-479] - SlaveRecoveryTest/0.CleanupExecutor failure.
* [MESOS-485] - Latest trunk fails on strict aliasing on CentOS
+ * [MESOS-488] - Master should not send FrameworkError messages if the same scheduler tries to re-register
* [MESOS-490] - Update mesos-daemon.sh (and associated scripts) to work with new flags mechanisms.
* [MESOS-497] - Queued tasks should be launched in the order they were received
* [MESOS-499] - Local slave run crashes on startup
@@ -66,6 +68,14 @@ Release Notes - Mesos - Version 0.14.0
* [MESOS-644] - Slave doesn't correctly handle checkpointed terminal update whose ack doesn't reach the executor
* [MESOS-646] - Slave recovery doesn't properly handle checkpointed queued tasks
* [MESOS-648] - Slave should properly handle partial writes of status updates
+ * [MESOS-657] - SlaveRecoveryTest/1.PartitionedSlave fails with cgroups
+ * [MESOS-658] - A framework can be incorrectly removed by the Master.
+ * [MESOS-668] - SlaveRecoveryTest/0.MultipleFrameworks flaky
+ * [MESOS-671] - CgroupsIsolator does not listen for OOM events on recovered executors.
+ * [MESOS-673] - Task reconciliation does not properly release executor resources.
+ * [MESOS-675] - CHECK failure in the Master.
+ * [MESOS-676] - Slave::reregistered LOG(FATAL)s due to being in RECOVERING state.
+ * [MESOS-689] - Master incorrectly rejects tasks for long lived executors if they don't have FrameworkID set
** Improvement
* [MESOS-179] - Need to check for Python development headers
@@ -82,6 +92,9 @@ Release Notes - Mesos - Version 0.14.0
* [MESOS-613] - Give better guidance when recovery fails
* [MESOS-626] - Add the ability for example frameworks to checkpoint
* [MESOS-634] - Make slave recovery more robust by ignoring absence of files
+ * [MESOS-651] - Expose slave re-registration time in the Web UI
+ * [MESOS-663] - Expose recovery errors when running recovery in --no-strict mode
+ * [MESOS-666] - Include CHANGELOG and docs/ in distributions.
** New Feature
* [MESOS-110] - Slave Recovery: A slave restart should not restart tasks
http://git-wip-us.apache.org/repos/asf/mesos/blob/4cb7b74d/docs/Configuration.md
----------------------------------------------------------------------
diff --git a/docs/Configuration.md b/docs/Configuration.md
index f07a0b0..77e507c 100644
--- a/docs/Configuration.md
+++ b/docs/Configuration.md
@@ -10,15 +10,18 @@ Configuration values are searched for first in the environment, then on the comm
## Important Options ##
-These are only a subset of the options. The following list may not match the version of mesos you are running, as the flags evolve from time to time in the code.
+These are only a subset of the options. The following list may not match the version of mesos you are running, as the flags evolve from time to time in the code.
**For now, the definitive source for which flags your version of Mesos supports can be found by running the binary with the flag `--help`, for example `bin/mesos-master --help`**. This will change as we add canonical documentation for each release.
### Master and Slave Options ###
- `log_dir` : Directory for log files. If unspecified, nothing is written to disk.
+
- `quiet` : Disable logging to stderr. [Default: false]
+
- `ip` : IP address to listen on. [Optional]
+
- `port` : Port to listen on. [Default for master: 5050, Default for slave:5051]
### Master Options ###
@@ -28,17 +31,51 @@ These are only a subset of the options. The following list may not match the ver
- zk://username:password@host1:port1,host2:port2,.../path
- file://path/to/file (where file contains one of the above)
+ - `roles` : A comma seperated list of the allocation roles
+ that frameworks in this cluster may belong to.
+
+ - `weights` : A comma seperated list of role/weight pairs
+ of the form 'role=weight,role=weight'. Weights
+ are used to indicate forms of priority.
+
+**For the complete list of master options: ./mesos-master.sh --help**
+
### Slave Options ###
- - `resources` : Total consumable resources for the slave.
- - Ex: "cpus:8.0;memory:4096;disk:20480;ports:[30000-50000]"
+ - `resources` : Total consumable resources per slave, in
+ the form 'name(role):value;name(role):value...'.
+ - NOTE: '(role)' is optional.
+ - Ex: "cpus(role2):2;mem(role2):1024;cpus:1;mem:1024;disk:0"
+
- `attributes` : Attributes for the machine. [Optional]
- These are free-form in the same style as resources, ex: "rack:abc;kernel:2.6.44".
- This information is provided to frameworks.
- - `work_dir` : Directory for the framework work directories. [Default: /tmp/mesos]
+
+ - `work_dir` : Directory for the executor work directories. [Default: /tmp/mesos]
+
- `isolation` : Isolation mechanism, one of: "process", "cgroups". [Default: process]
+
- `cgroups_enable_cfs` : If using cgroups isolation, this enables hard limits on CPU resources.
+
- `master` : May be one of:
- zk://host1:port1,host2:port2,.../path
- zk://username:password@host1:port1,host2:port2,.../path\n
- file://path/to/file (where file contains one of the above)
+
+ - `default_role` : Any resources in the --resources flag that
+ omit a role, as well as any resources that
+ are not present in --resources but that are
+ automatically detected, will be assigned to
+ this role. [Default: *]
+
+ - `checkpoint` : Whether to checkpoint slave and frameworks information
+ to disk.
+ - This enables a restarted slave to recover status updates and reconnect
+ with (--recover=reconnect) or kill (--recover=kill) old executors [Default: false]
+
+ - `strict` : Whether to do recovery in strict mode [Default: true].
+ - If strict=true, any and all recovery errors are considered fatal.
+ - If strict=false, any errors (e.g., corruption in checkpointed data) during recovery are
+ ignored and as much state as possible is recovered.
+
+**For the complete list of slave options: ./mesos-slave.sh --help**
http://git-wip-us.apache.org/repos/asf/mesos/blob/4cb7b74d/docs/Mesos-developers-guide.md
----------------------------------------------------------------------
diff --git a/docs/Mesos-developers-guide.md b/docs/Mesos-developers-guide.md
index 57dfe28..f11c1e3 100644
--- a/docs/Mesos-developers-guide.md
+++ b/docs/Mesos-developers-guide.md
@@ -19,37 +19,36 @@
2. Most of your changes will probably be to files inside of <BASE_MESOS_DIR>/src
2. To build, we recommend that you don't build inside of the src directory. We recommend you do the following:
3. From inside of the root Mesos directory: `mkdir build && cd build`
- 3. `../configure.<TEMPLATE_SUFFIX_FOR_YOUR_MACHINE_AND_ENV>` (e.g. `configure.macosx`)
+ 3. `../configure`
3. `make`
- 3. now all of the files generated by the build process will be contained in the build directory you created, instead of being spread throughout the src directory, which is a bit messier. This is both cleaner, and makes it easy to clean up if you want to get rid of the files generated by `configure` and `make`. I.e. You can reset your build process without risking changes you made in the src directory, by simply deleting the build directory, and creating a new one.
+ 3. Now all of the files generated by the build process will be contained in the build directory you created, instead of being spread throughout the src directory, which is a bit messier. This is both cleaner, and makes it easy to clean up if you want to get rid of the files generated by `configure` and `make`. I.e. You can reset your build process without risking changes you made in the src directory, by simply deleting the build directory, and creating a new one.
1. Make sure all of your test cases now pass.
+ 2. `make check`
1. Think you're code is ready for review?
2. Make sure to pull in any changes that have been committed to trunk. If you are using Git, do this via something like:
- 3. `git remote update`
- 3. `git rebase origin/trunk`
- 3. Check the output of `git diff origin/trunk` and make sure it lists only your changes. If other changes you did not make are listed, try a merge to bring your branch up to date with origin/trunk.
- 2. Make sure it is building and all test cases are passing (run bin/tests/all-tests). To be extra sure, before moving on to requesting a review request, we recommend running `make clean && make` and then running the test cases.
-
-1. Create a diff of your code changes, including your test case.
- 2. If you are using Git, and you haven't committed any of your changes to your local clone of the repository, use the command: `git diff > MESOS-XX.patch` where XX is your issue number
- 2. If you are using SVN, use the command: `svn diff > MESOS-XX.patch`
-
-1. Create a <i>review request</i> at Review Board (RB), http://reviews.apache.org, and attach your diff. More detailed instructions:
- 2. Log in or create an account at reviews.apache.org (Apache's infrastructure doesn't have any sort of single-sign-in process, so you need to have an account specific to this instance of RB)
- 2. Create a new review request at https://reviews.apache.org/r/new/
- 3. Under the field labeled "Repository" choose "Mesos"
- 3. Leave the field labeled "Base Directory" blank
- 3. Upload your diff via the field labeled "Diff"
- 3. Be sure to add your JIRA issue id (e.g. MESOS-01) to the field labeled "Bugs" (this will automatically link)
- 3. Under "Description" in addition to details about your changes, include a description of any wiki documentation pages need to be added, or are affected by your changes (e.g. did you change or add any configuration options/flags? Did you add a new binary?)
+ 3. `git checkout master`
+ 3. `git pull`
+ 4. `git co my_branch`
+ 3. Check the output of `git diff master` and make sure it lists only your changes. If other changes you did not make are listed, try a rebase to bring your branch up to date with master.
+ 2. Make sure it is building and all test cases are passing by running `make clean && make check`.
+
+1. Detailed instructions for posting a review:
+ 2. Log in or create an account at reviews.apache.org (Apache's infrastructure doesn't have any sort of single-sign-in process, so you need to have an account specific to this instance of RB)
+ 2. The easiest (and recommended) way to submit reviews is through `post-reviews.py` a wrapper around post-review.
+ 2. First, install post-review. [See Instructions](http://www.reviewboard.org/docs/manual/dev/users/tools/post-review/)
+ 2. From your local branch run `support/post-reviews.py`.
+ 2. Note that `post-reviews.py` creates a new review for every commit on your branch that is different from the `master`.
+ 2. Be sure to add your JIRA issue id (e.g. MESOS-01) to the field labeled "Bugs" (this will automatically link)
+ 2. Under "Description" in addition to details about your changes, include a description of any wiki documentation pages need to be added, or are affected by your changes (e.g. did you change or add any configuration options/flags? Did you add a new binary?)
1. Wait for a code review from another Mesos developer via Review Board, address their feedback and upload updated patches until you receive a "Ship It" from a Mesos committer.
2. Review Board comments should be used for code-specific discussions, and JIRA comments for bigger-picture design discussions.
2. Always respond to each RB comment that you address directly (i.e. each comment can be responded to directly) with either "Done." or a comment explaining how you addressed it.
+ 2. If an issue has been raised in the review, please resolve the issue as "Fixed" or "Dropped". If "Dropped" please add a comment explaining the reason. Also, if your fix warrants a comment (e.g., fixed differently than suggested by the reviewer) please add a comment.
-1. After consensus is reached on your JIRA/patch, you're review request will receive a "Ship It!" from a committer, and then a committer will commit your patch to the SVN repository. Congratulations and thanks for participating in our community!
+1. After consensus is reached on your JIRA/patch, you're review request will receive a "Ship It!" from a committer, and then a committer will commit your patch to the git repository. Congratulations and thanks for participating in our community!
1. Please ensure that the necessary wiki documentation gets created or updated (i.e. make the changes yourself!)
@@ -59,7 +58,8 @@
* Always assign the JIRA to yourself before you start working on it. This helps to avoid duplication of work
## Using Review Board
+* We use [Apache Review Board](https://reviews.apache.org) to host reviews.
* A code review request should be created for every JIRA that involves a change to the codebase.
## Style Guides
-* [Mesos C++ Style Guide](Mesos-c++-style-guide.md)
\ No newline at end of file
+* [Mesos C++ Style Guide](Mesos-c++-style-guide.md)
http://git-wip-us.apache.org/repos/asf/mesos/blob/4cb7b74d/docs/Running-Hadoop-on-Mesos.md
----------------------------------------------------------------------
diff --git a/docs/Running-Hadoop-on-Mesos.md b/docs/Running-Hadoop-on-Mesos.md
deleted file mode 100644
index c128e4a..0000000
--- a/docs/Running-Hadoop-on-Mesos.md
+++ /dev/null
@@ -1,33 +0,0 @@
-We have ported version 0.20.205.0 of Hadoop to run on Mesos. Most of the Mesos port is implemented by a pluggable Hadoop scheduler, which communicates with Mesos to receive nodes to launch tasks on. However, a few small additions to Hadoop's internal APIs are also required.
-
-You can build the ported version of Hadoop using `make hadoop`. It gets placed in the `hadoop/hadoop-0.20.205.0` directory. However, if you want to patch your own version of Hadoop to add Mesos support, you can also use .patch files located in `<Mesos directory>/hadoop`. These patches are likely to work on other Hadoop versions derived from 0.20. For example, for Cloudera's Distribution, GitHub user patelh has already created a [Mesos-compatible version of CDH3u3](https://github.com/patelh/cdh3u3-with-mesos).
-
-To run Hadoop on Mesos, follow these steps:
-<ol>
-<li> Run `make hadoop` to build Hadoop 0.20.205.0 with Mesos support, or `TUTORIAL.sh` to patch and build your own Hadoop version.</li>
-<li> Set up [Hadoop's configuration](http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster) as you would usually do with a new install of Hadoop, following the [instructions on the Hadoop website](http://hadoop.apache.org/common/docs/r0.20.2/index.html) (at the very least, you need to set <code>JAVA_HOME</code> in Hadoop's <code>conf/hadoop-env.sh</code> and set <code>mapred.job.tracker</code> in <code>conf/mapred-site.xml</code>).</li>
-</li>
-<li> Add the following parameters to Hadoop's <code>conf/mapred-site.xml</code>:
-<pre>
-<property>
- <name>mapred.jobtracker.taskScheduler</name>
- <value>org.apache.hadoop.mapred.MesosScheduler</value>
-</property>
-<property>
- <name>mapred.mesos.master</name>
- <value>[URL of Mesos master]</value>
-</property>
-</pre>
-</li>
-<li> Launch a JobTracker with <code>bin/hadoop jobtracker</code> (<i>do not</i> use <code>bin/start-mapred.sh</code>). The JobTracker will then launch TaskTrackers on Mesos when jobs are submitted.</li>
-<li> Submit jobs to your JobTracker as usual.</li>
-</ol>
-
-Note that when you run on a cluster, Hadoop (and Mesos) should be located on the same path on all nodes.
-
-If you wish to run multiple JobTrackers, the easiest way is to give each one a different port by using a different Hadoop `conf` directory for each one and passing the `--conf` flag to `bin/hadoop` to specify which config directory to use. You can copy Hadoop's existing `conf` directory to a new location and modify it to achieve this.
-
-## Hadoop Versions with Mesos Support Available
-
-* 0.20.205.0: Included in Mesos (as described above).
-* CDH3u3: [https://github.com/patelh/cdh3u3-with-mesos](https://github.com/patelh/cdh3u3-with-mesos)
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/mesos/blob/4cb7b74d/docs/Running-a-Second-Instance-of-Hadoop-(Snow-Leopard).md
----------------------------------------------------------------------
diff --git a/docs/Running-a-Second-Instance-of-Hadoop-(Snow-Leopard).md b/docs/Running-a-Second-Instance-of-Hadoop-(Snow-Leopard).md
deleted file mode 100644
index ba57f97..0000000
--- a/docs/Running-a-Second-Instance-of-Hadoop-(Snow-Leopard).md
+++ /dev/null
@@ -1,26 +0,0 @@
-1. First, follow the instructions [here](https://github.com/mesos/mesos/wiki/Running-Mesos-On-Mac-OS-X-Snow-Leopard-(Single-Node-Cluster\)) to get a single instance of Hadoop running.
-
-2. Next, we'll get a second instance of the same version of Hadoop that ships with Mesos running:
- - Make another copy of Hadoop:
- `~/mesos$ cp -R frameworks/hadoop-0.20.2 ~/hadoop`
- - Modify necessary ports:
- * In conf/mapred-site.xml.template, change the mapred.job.tracker port from 9001 to 9002.
- * In src/mapred/mapred-default.xml, change the mapred.task.tracker.http.address port and the
- mapred.job.tracker.http.address port both to 0.
- - Build Hadoop:
- `~/hadoop$ ant`
- `~/hadoop$ ant compile-core jar`
- `~/hadoop$ ant examples jar`
- - Start up Mesos:
- `~/mesos$ bin/mesos-master`
- `~/mesos$ bin/mesos-slave --master=mesos://master@localhost:5050`
- - Start up HDFS (we'll have one instance of HDFS that both instances of Hadoop access):
- `~/mesos/frameworks/hadoop-0.20.2$ bin/hadoop namenode`
- `~/mesos/frameworks/hadoop-0.20.2$ bin/hadoop datanode`
- - Start the jobtrackers:
- `~/mesos/frameworks/hadoop-0.20.2$ bin/hadoop jobtracker`
- `~/hadoop$ bin/hadoop jobtracker`
- - Run some tests:
- `~/mesos/frameworks/hadoop-0.20.2$ bin/hadoop jar build/hadoop-0.20.3-dev-examples.jar wordcount ~/gutenburg/ ~/output`
- `~/hadoop$ bin/hadoop jar build/hadoop-0.20.3-dev-examples.jar wordcount ~/gutenburg/ ~/output1`
- - At this point, you should be able to run both instances of wordcount at the same time.
\ No newline at end of file