You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mesos.apache.org by be...@apache.org on 2013/03/03 20:38:26 UTC
svn commit: r1452108 - in /incubator/mesos/trunk/docs: ./ images/
Author: benh
Date: Sun Mar 3 19:38:25 2013
New Revision: 1452108
URL: http://svn.apache.org/r1452108
Log:
Fixed all links to user proper Markdown or Textile syntax.
Added:
incubator/mesos/trunk/docs/images/
incubator/mesos/trunk/docs/images/architecture-example.jpg
incubator/mesos/trunk/docs/images/architecture3.jpg
Modified:
incubator/mesos/trunk/docs/90-day-plan.md
incubator/mesos/trunk/docs/Allocation-module.textile
incubator/mesos/trunk/docs/Configuration.textile
incubator/mesos/trunk/docs/DRAFT:-Running-Hadoop-on-Mesos-UPDATED-INCOMPLETELY-12-13-11.md
incubator/mesos/trunk/docs/Deploy-Scripts.textile
incubator/mesos/trunk/docs/EC2-Scripts.textile
incubator/mesos/trunk/docs/Event-history.md
incubator/mesos/trunk/docs/Home.md
incubator/mesos/trunk/docs/Logging-and-Debugging.textile
incubator/mesos/trunk/docs/Mesos-Architecture.md
incubator/mesos/trunk/docs/Mesos-Code-Internals.textile
incubator/mesos/trunk/docs/Mesos-Roadmap.md
incubator/mesos/trunk/docs/Mesos-configure-command-flag-options.md
incubator/mesos/trunk/docs/Mesos-developers-guide.md
incubator/mesos/trunk/docs/Mesos-ready-to-go-AMI.md
incubator/mesos/trunk/docs/Old-mesos-build-instructions.md
incubator/mesos/trunk/docs/Running-Hadoop-on-Mesos.md
incubator/mesos/trunk/docs/Running-Mesos-On-Mac-OS-X-Snow-Leopard-(Single-Node-Cluster).md
incubator/mesos/trunk/docs/Running-a-web-application-farm-on-mesos.textile
incubator/mesos/trunk/docs/Using-Linux-Containers.textile
Modified: incubator/mesos/trunk/docs/90-day-plan.md
URL: http://svn.apache.org/viewvc/incubator/mesos/trunk/docs/90-day-plan.md?rev=1452108&r1=1452107&r2=1452108&view=diff
==============================================================================
--- incubator/mesos/trunk/docs/90-day-plan.md (original)
+++ incubator/mesos/trunk/docs/90-day-plan.md Sun Mar 3 19:38:25 2013
@@ -9,7 +9,7 @@
# Details
-This 90-day plan aims to bolster the Mesos community. It addresses "Growing a development community and promoting adoption" in the [[Mesos Roadmap]].
+This 90-day plan aims to bolster the Mesos community. It addresses "Growing a development community and promoting adoption" in the [Mesos Roadmap](Mesos-Roadmap.md).
Start: March 1st, 2013. End: May 30, 2013.
Modified: incubator/mesos/trunk/docs/Allocation-module.textile
URL: http://svn.apache.org/viewvc/incubator/mesos/trunk/docs/Allocation-module.textile?rev=1452108&r1=1452107&r2=1452108&view=diff
==============================================================================
--- incubator/mesos/trunk/docs/Allocation-module.textile (original)
+++ incubator/mesos/trunk/docs/Allocation-module.textile Sun Mar 3 19:38:25 2013
@@ -1,6 +1,6 @@
h1. Overview
-The logic that the Mesos master uses to determine which frameworks to make offer resource offers to is encapsulated in the Master's _allocation module_. The allocation module is a pluggable component that organizations can use to implement their own sharing policy, e.g. fair-sharing, Dominant Resource Fairness (see [[the DRF paper|http://www.eecs.berkeley.edu/Pubs/TechRpts/2010/EECS-2010-55.pdf]]), priority, etc.
+The logic that the Mesos master uses to determine which frameworks to make offer resource offers to is encapsulated in the Master's _allocation module_. The allocation module is a pluggable component that organizations can use to implement their own sharing policy, e.g. fair-sharing, Dominant Resource Fairness (see "the DRF paper":http://www.eecs.berkeley.edu/Pubs/TechRpts/2010/EECS-2010-55.pdf), priority, etc.
h1. Allocation Module API
Modified: incubator/mesos/trunk/docs/Configuration.textile
URL: http://svn.apache.org/viewvc/incubator/mesos/trunk/docs/Configuration.textile?rev=1452108&r1=1452107&r2=1452108&view=diff
==============================================================================
--- incubator/mesos/trunk/docs/Configuration.textile (original)
+++ incubator/mesos/trunk/docs/Configuration.textile Sun Mar 3 19:38:25 2013
@@ -17,7 +17,7 @@ The following options are commonly confi
|_. Parameter |_. Description|
|@webui_port@ | Port to bind to for user-viewable web UI. |
-|@log_dir@ | Directory to place logs into, including [[event history]] logs. It is recommended to use a local disk for each node. |
+|@log_dir@ | Directory to place logs into, including "event history":Event-history.md logs. It is recommended to use a local disk for each node. |
|@quiet@ | Disable logging to standard error. |
|@conf@ | Specifies a config directory to use instead of the default one. |
@@ -27,13 +27,13 @@ The following options are commonly confi
|@master@ | Master URL to connect to. |
|@resources@ | A semicolon separated set of _key:value_ pairs, e.g. <code>cpus:1;mem:1024</code> |
|@work_dir@ | Directory for slaves to place framework's output files in. It is recommended to use a local disk for each slave. |
-|@isolation@ | Isolation module to use for isolating tasks. The default is @process@, which performs no isolation other than running each framework as its user, but @lxc@ can be used on systems that support Linux Containers. See [[using Linux containers]] for details. |
+|@isolation@ | Isolation module to use for isolating tasks. The default is @process@, which performs no isolation other than running each framework as its user, but @lxc@ can be used on systems that support Linux Containers. See "using Linux containers":Using-Linux-Containers.textile for details. |
|@hadoop_home@ | Location of Hadoop, if you wish to use the Hadoop Distributed File System for distributing framework binaries across the cluster. |
The following options are commonly configured for the master:
|_. Parameter |_. Description|
-|@url@ | A ZooKeeper URL can be specified when [[using ZooKeeper]]. |
+|@url@ | A ZooKeeper URL can be specified when "using ZooKeeper":Using-ZooKeeper.textile. |
|@port@ | Port to bind to for communication between master and slaves. |
-|@allocator@ | Which pluggable [[allocation module]] to use. Currently there is only one, @simple@, that performs fair sharing between frameworks. |
+|@allocator@ | Which pluggable "allocation module":Allocation-module.textile to use. Currently there is only one, @simple@, that performs fair sharing between frameworks. |
|@ip@ | Which IP address to bind to (to make the master bind to a desired network interface if it does not do so by default). |
Modified: incubator/mesos/trunk/docs/DRAFT:-Running-Hadoop-on-Mesos-UPDATED-INCOMPLETELY-12-13-11.md
URL: http://svn.apache.org/viewvc/incubator/mesos/trunk/docs/DRAFT%3A-Running-Hadoop-on-Mesos-UPDATED-INCOMPLETELY-12-13-11.md?rev=1452108&r1=1452107&r2=1452108&view=diff
==============================================================================
--- incubator/mesos/trunk/docs/DRAFT:-Running-Hadoop-on-Mesos-UPDATED-INCOMPLETELY-12-13-11.md (original)
+++ incubator/mesos/trunk/docs/DRAFT:-Running-Hadoop-on-Mesos-UPDATED-INCOMPLETELY-12-13-11.md Sun Mar 3 19:38:25 2013
@@ -5,7 +5,7 @@ The ported version of Hadoop is included
To run Hadoop on Mesos, follow these steps:
<ol>
<li> Build Hadoop using <code>ant</code>.</li>
-<li> Set up [[Hadoop's configuration|http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/]] as you would usually do with a new install of Hadoop, following the [[instructions on the Hadoop website|http://hadoop.apache.org/common/docs/r0.20.2/index.html]] (at the very least, you need to set <code>JAVA_HOME</code> in Hadoop's <code>conf/hadoop-env.sh</code> and set <code>mapred.job.tracker</code> in <code>conf/mapred-site.xml</code>).</li>
+<li> Set up [Hadoop's configuration](http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster) as you would usually do with a new install of Hadoop, following the [instructions on the Hadoop website](http://hadoop.apache.org/common/docs/r0.20.2/index.html) (at the very least, you need to set <code>JAVA_HOME</code> in Hadoop's <code>conf/hadoop-env.sh</code> and set <code>mapred.job.tracker</code> in <code>conf/mapred-site.xml</code>).</li>
</li>
<li> Add the following parameters to Hadoop's <code>conf/mapred-site.xml</code>:
<pre>
Modified: incubator/mesos/trunk/docs/Deploy-Scripts.textile
URL: http://svn.apache.org/viewvc/incubator/mesos/trunk/docs/Deploy-Scripts.textile?rev=1452108&r1=1452107&r2=1452108&view=diff
==============================================================================
--- incubator/mesos/trunk/docs/Deploy-Scripts.textile (original)
+++ incubator/mesos/trunk/docs/Deploy-Scripts.textile Sun Mar 3 19:38:25 2013
@@ -2,7 +2,7 @@ Mesos includes a set of scripts in @MESO
It is also possible to set environment variables, ulimits, etc that will affect the master and slave. by editing @MESOS_HOME/deploy/mesos-env.sh@. One particularly useful setting is @LIBPROCESS_IP@, which tells the master and slave binaries which IP address to bind to; in some installations, the default interface that the hostname resolves to is not the machine's external IP address, so you can set the right IP through this variable.
-Finally, the deploy scripts do not use ZooKeeper by default. If you want to use ZooKeeper (for multiple masters), you can do so by either editing @deploy/mesos-env.sh@ to set @MESOS_URL@ to a @zoo://@ or @zoofile://@ URL, or by editing @conf/mesos.conf@ to set the @url@ configuration parameter. Please see [[Using ZooKeeper]] for details.
+Finally, the deploy scripts do not use ZooKeeper by default. If you want to use ZooKeeper (for multiple masters), you can do so by either editing @deploy/mesos-env.sh@ to set @MESOS_URL@ to a @zoo://@ or @zoofile://@ URL, or by editing @conf/mesos.conf@ to set the @url@ configuration parameter. Please see "Using ZooKeeper":Using-ZooKeeper.textile for details.
h1. Notes
* The deploy scripts assume that Mesos is located in the same directory on all nodes.
Modified: incubator/mesos/trunk/docs/EC2-Scripts.textile
URL: http://svn.apache.org/viewvc/incubator/mesos/trunk/docs/EC2-Scripts.textile?rev=1452108&r1=1452107&r2=1452108&view=diff
==============================================================================
--- incubator/mesos/trunk/docs/EC2-Scripts.textile (original)
+++ incubator/mesos/trunk/docs/EC2-Scripts.textile Sun Mar 3 19:38:25 2013
@@ -30,7 +30,7 @@ h1. Running Jobs
* If your job needs to access large datasets, the fastest way to do that is to load them from Amazon S3 or an Amazon EBS device into an instance of the Hadoop Distributed File System (HDFS) on your nodes. The @mesos-ec2@ script already sets up a HDFS instance for you. It's installed in @/root/ephemeral-hdfs@, and can be accessed using the @bin/hadoop@ script in that directory. Note that the data in this HDFS goes away when you stop and restart a machine.
* There is also a _persistent HDFS_ instance in @/root/presistent-hdfs@ that will keep data across cluster restarts. Typically each node has relatively little space of persistent data (about 3 GB), but you can use the @--ebs-vol-size@ option to @mesos-ec2@ to attach a persistent EBS volume to each node for storing the persistent HDFS.
-If you get an "Executor on slave X disconnected" error when running your framework, you probably haven't copied your code the slaves. Use the @~/mesos-ec2/copy-dir@ script to do that. If you keep getting the error, though, look at the slave's logs for that framework using the Mesos web UI. Please see [[logging and debugging]] for details.
+If you get an "Executor on slave X disconnected" error when running your framework, you probably haven't copied your code the slaves. Use the @~/mesos-ec2/copy-dir@ script to do that. If you keep getting the error, though, look at the slave's logs for that framework using the Mesos web UI. Please see "logging and debugging":Logging-and-Debugging.textile for details.
h1. Terminating a Cluster
@@ -49,4 +49,4 @@ The @mesos-ec2@ script also supports pau
h1. Limitations
-* The @mesos-ec2@ script currently does not use the [[deploy scripts]] included with Mesos to manage its clusters. This will likely be fixed in the future.
\ No newline at end of file
+* The @mesos-ec2@ script currently does not use the "deploy scripts":Deploy-Scripts.textile included with Mesos to manage its clusters. This will likely be fixed in the future.
\ No newline at end of file
Modified: incubator/mesos/trunk/docs/Event-history.md
URL: http://svn.apache.org/viewvc/incubator/mesos/trunk/docs/Event-history.md?rev=1452108&r1=1452107&r2=1452108&view=diff
==============================================================================
--- incubator/mesos/trunk/docs/Event-history.md (original)
+++ incubator/mesos/trunk/docs/Event-history.md Sun Mar 3 19:38:25 2013
@@ -1,3 +1,3 @@
-This is a feature [[currently being developed|https://github.com/mesos/mesos/tree/andyk-event-history-nowebui]] that will allow capture the history of significant events in the context of all things Mesos (e.g. Jobs/tasks registering, starting, failing, etc.) into a variety of places where it can be used in the future.
+This is a feature [currently being developed](https://github.com/mesos/mesos/tree/andyk-event-history-nowebui) that will allow capture the history of significant events in the context of all things Mesos (e.g. Jobs/tasks registering, starting, failing, etc.) into a variety of places where it can be used in the future.
This is still under development.
\ No newline at end of file
Modified: incubator/mesos/trunk/docs/Home.md
URL: http://svn.apache.org/viewvc/incubator/mesos/trunk/docs/Home.md?rev=1452108&r1=1452107&r2=1452108&view=diff
==============================================================================
--- incubator/mesos/trunk/docs/Home.md (original)
+++ incubator/mesos/trunk/docs/Home.md Sun Mar 3 19:38:25 2013
@@ -45,7 +45,7 @@ To get running with Mesos version 0.9.0-
1. [Download Mesos 0.9.0-incubating](http://www.apache.org/dyn/closer.cgi/incubator/mesos/mesos-0.9.0-incubating/)
1. Run configure (there are a few different helper scripts that wrap the `configure` script called configure.<type-of-os>)
1. In OS X: run `./configure.macosx`. If you're running Mountain Lion, you may need to follow the instructions [here](https://issues.apache.org/jira/browse/MESOS-261?focusedCommentId=13447058&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13447058) and [here](https://issues.apache.org/jira/browse/MESOS-285).
- 1. In Linux, you can probably just run `./configure --with-webui --with-included-zookeeper`. These flags are what we recommend; advanced users may want to exclude these flags or use others, see [[Mesos Configure Command Flag Options]].
+ 1. In Linux, you can probably just run `./configure --with-webui --with-included-zookeeper`. These flags are what we recommend; advanced users may want to exclude these flags or use others, see [Mesos Command-Line Flags](Configuration.textile).
1. run `make`
### NOTES:
@@ -82,29 +82,28 @@ After you have done this, you can also s
To run Mesos on more than one machine, you have multiple options:
-* To launch a cluster on a private cluster that you own, use Mesos's [[deploy scripts]]
-* To launch a cluster on Amazon EC2, you can use the Mesos [[EC2 scripts]]
+* To launch a cluster on a private cluster that you own, use Mesos's [deploy scripts](Deploy-Scripts.textile)
+* To launch a cluster on Amazon EC2, you can use the Mesos [EC2 scripts](EC2-Scripts.textile)
# Where to Go From Here
-* [[Mesos architecture]] -- an overview of Mesos concepts.
-* [[Mesos Developers guide]] -- resources for developers contributing to Mesos. Style guides, tips for working with SVN/git, and more!
-* [[App/Framework development guide]] -- learn how to build applications on top of Mesos.
-* [[Configuring Mesos|Configuration]] -- a guide to the various settings that can be passed to Mesos daemons.
+* [Mesos Architecture](Mesos-Architecture.md) -- an overview of Mesos concepts.
+* [Mesos Developers Guide](Mesos-developers-guide.md) -- resources for developers contributing to Mesos. Style guides, tips for working with SVN/git, and more!
+* [App/Framework Development Guide](App-Framework-development-guide.textile) -- learn how to build applications on top of Mesos.
+* [Mesos Command-Line Flags](Configuration.textile) -- a guide to the various flags that can be set.
* Mesos system feature guides:
- * [[Deploy scripts]] for launching a Mesos cluster on a set of machines.
- * [[EC2 scripts]] for launching a Mesos cluster on Amazon EC2.
- * [[Logging and Debugging]] -- viewing Mesos and framework logs.
- * [[Using ZooKeeper]] for master fault-tolerance.
- * [[Using Linux Containers]] for resource isolation on slaves.
+ * [Deploy Scripts](Deploy-Scripts.textile) for launching a Mesos cluster on a set of machines.
+ * [EC2 scripts](EC2-Scripts.textile) for launching a Mesos cluster on Amazon EC2.
+ * [Logging and Debugging](Logging-and-Debugging.textile) -- viewing Mesos and framework logs.
+ * [Using ZooKeeper](Using-ZooKeeper.textile) for master fault-tolerance.
+ * [Using Linux Containers](Using-Linux-Containers.textile) for resouce isolation on slaves.
* Guide to running existing frameworks:
- * [[Running Spark on Mesos|https://github.com/mesos/spark/wiki]]
- * [[Using the Mesos Submit tool]]
- * [[Using Mesos with Hypertable on EC2|http://code.google.com/p/hypertable/wiki/Mesos]] (external link) - Hypertable is a distributed database platform.
- * [[Running Hadoop on Mesos]]
- * [[Running a web application farm on Mesos]]
- * [[Running Torque or MPI on Mesos]]
-* [[Powered-by|Powered-by Mesos]] -- Projects that are using Mesos!
-* [[Mesos code internals overview|Mesos Code Internals]] -- an overview of the codebase and internal organization.
-* [[Mesos development roadmap|Mesos Roadmap]]
-* [[The official Mesos website|http://mesosproject.org]]
\ No newline at end of file
+ * [Running Spark on Mesos](https://github.com/mesos/spark/wiki)
+ * [Using the Mesos Submit tool](Using-the-mesos-submit-tool.md)
+ * [Using Mesos with Hypertable on EC2](http://code.google.com/p/hypertable/wiki/Mesos) - Hypertable is a distributed database platform.
+ * [Running Hadoop on Mesos](Running-Hadoop-on-Mesos.md)
+ * [Running a web application farm on Mesos](Running-a-web-application-farm-on-mesos.textile)
+ * [Running Torque or MPI on Mesos](Running-torque-or-mpi-on-mesos.md)
+* [Powered-by Mesos](Powered-by-Mesos.md) -- Projects that are using Mesos!
+* [Mesos Code Internals](Mesos-Code-Internals.textile) -- an overview of the codebase and internal organization.
+* [Mesos Development Roadmap](Mesos-Roadmap.md)
Modified: incubator/mesos/trunk/docs/Logging-and-Debugging.textile
URL: http://svn.apache.org/viewvc/incubator/mesos/trunk/docs/Logging-and-Debugging.textile?rev=1452108&r1=1452107&r2=1452108&view=diff
==============================================================================
--- incubator/mesos/trunk/docs/Logging-and-Debugging.textile (original)
+++ incubator/mesos/trunk/docs/Logging-and-Debugging.textile Sun Mar 3 19:38:25 2013
@@ -1,5 +1,5 @@
-Mesos uses the "Google Logging library":http://code.google.com/p/google-glog and writes logs to @MESOS_HOME/logs@ by default, where @MESOS_HOME@ is the location where Mesos is installed. The log directory can be [[configured|Configuration]] using the @log_dir@ parameter.
+Mesos uses the "Google Logging library":http://code.google.com/p/google-glog and writes logs to @MESOS_HOME/logs@ by default, where @MESOS_HOME@ is the location where Mesos is installed. The log directory can be "configured":Configuration.textile using the @log_dir@ parameter.
Frameworks that run on Mesos have their output stored to a "work" directory on each machine. By default, this is @MESOS_HOME/work@. Within this directory, a framework's output is placed in files called @stdout@ and @stderr@ in a directory of the form @slave-X/fw-Y/Z@, where X is the slave ID, Y is the framework ID, and multiple subdirectories Z are created for each attempt to run an executor for the framework. These files can also be accessed via the web UI of the slave daemon.
-Finally, using an [[alpha code branch|https://github.com/mesos/mesos/tree/andyk-event-history-nowebui]] which (at the time of this writing) is still in an alpha state, structured logs of cluster activity can be written to a tab-separated text file or a SQLite database using the [[event history]] system.
+Finally, using an "alpha code branch":https://github.com/mesos/mesos/tree/andyk-event-history-nowebui which (at the time of this writing) is still in an alpha state, structured logs of cluster activity can be written to a tab-separated text file or a SQLite database using the "event history":Event-history.md system.
Modified: incubator/mesos/trunk/docs/Mesos-Architecture.md
URL: http://svn.apache.org/viewvc/incubator/mesos/trunk/docs/Mesos-Architecture.md?rev=1452108&r1=1452107&r2=1452108&view=diff
==============================================================================
--- incubator/mesos/trunk/docs/Mesos-Architecture.md (original)
+++ incubator/mesos/trunk/docs/Mesos-Architecture.md Sun Mar 3 19:38:25 2013
@@ -1,16 +1,16 @@
-<p align="center">[[/images/architecture3.jpg|height=300px]]</p>
+<p align="center"><img src="/images/architecture3.jpg" height="300px"></p>
The above figure shows the main components of Mesos. Mesos consists of a <i>master</i> daemons that manages <i>slave</i> daemons running on each cluster node, and <i>mesos applications</i> (also called <i>frameworks</i>) that run <i>tasks</i> on these slaves.
The master enables fine-grained sharing of resources (cpu, ram, ...) across applications by making them <i>resource offers</i>. Each resource offer contains a list of <slave ID, resource1: amount1, resource2, amount2, ...>. The master decides <i>how many</i> resources to offer to each framework according to a given organizational policy, such as fair sharing, or strict priority. To support a diverse set of policies, the master employs a modular architecture that makes it easy to add new allocation modules via a plugin mechanism.
-A framework running on top of Mesos consists of two components: a <i>scheduler</i> that registers with the master to be offered resources, and an <i>executor</i> process that is launched on slave nodes to run the framework's tasks (see the [[App/Framework development guide]] for more details about application schedulers and executors). While the master determines <b>how many</b> resources are offered to each framework, the frameworks' schedulers select <b>which</b> of the offered resources to use. When a frameworks accepts offered resources, it passes to Mesos a description of the tasks it wants to run on them. In turn, Mesos launches the tasks on the corresponding slaves.
+A framework running on top of Mesos consists of two components: a <i>scheduler</i> that registers with the master to be offered resources, and an <i>executor</i> process that is launched on slave nodes to run the framework's tasks (see the [App/Framework development guide](App-Framework-development-guide.textile) for more details about application schedulers and executors). While the master determines <b>how many</b> resources are offered to each framework, the frameworks' schedulers select <b>which</b> of the offered resources to use. When a frameworks accepts offered resources, it passes to Mesos a description of the tasks it wants to run on them. In turn, Mesos launches the tasks on the corresponding slaves.
## Example of resource offer
The figure below shows an example of how a framework gets scheduled to run a task.
-<p align="center">[[/images/architecture-example.jpg|height=300px]]</p>
+<p align="center"><img src="/images/architecture-example.jpg" height="300px"></p>
Let's walk through the events in the figure.
@@ -23,4 +23,4 @@ In addition, this resource offer process
While the thin interface provided by Mesos allows it to scale and allows the frameworks to evolve independently, one question remains: how can the constraints of a framework be satisfied without Mesos knowing about these constraints? For example, how can a framework achieve data locality without Mesos knowing which nodes store the data required by the framework? Mesos answers these questions by simply giving frameworks the ability to <b>reject</b> offers. A framework will reject the offers that do not satisfy its constraints and accept the ones that do. In particular, we have found that a simple policy called delay scheduling \cite{delay-scheduling}, in which frameworks wait for a limited time to acquire nodes storing the input data, yields nearly optimal data locality.
-You can also read much more about the Mesos architecture in this [[technical paper|http://mesos.berkeley.edu/mesos_tech_report.pdf]].
+You can also read much more about the Mesos architecture in this [technical paper](http://mesos.berkeley.edu/mesos_tech_report.pdf).
Modified: incubator/mesos/trunk/docs/Mesos-Code-Internals.textile
URL: http://svn.apache.org/viewvc/incubator/mesos/trunk/docs/Mesos-Code-Internals.textile?rev=1452108&r1=1452107&r2=1452108&view=diff
==============================================================================
--- incubator/mesos/trunk/docs/Mesos-Code-Internals.textile (original)
+++ incubator/mesos/trunk/docs/Mesos-Code-Internals.textile Sun Mar 3 19:38:25 2013
@@ -2,8 +2,8 @@ h1. Mesos Code Internals
h2. Top-level directories in the Mesos distribution
-* ec2 - scripts for launching a Mesos cluster on EC2. See the wiki page on [[EC2-Scripts]]
-* frameworks - Included Mesos Frameworks. See the READMEs in each one. See the [[App/Framework development guide]] for a crash course in how Mesos Frameworks get resources from the Mesos master.
+* ec2 - scripts for launching a Mesos cluster on EC2. See the wiki page on "EC2-Scripts":EC2-Scripts.textile
+* frameworks - Included Mesos Frameworks. See the READMEs in each one. See the "App/Framework development guide":App-Framework-development-guide.textile for a crash course in how Mesos Frameworks get resources from the Mesos master.
* include - Contains headers that contain the interfaces that Mesos users need in order to interact with Mesos (e.g. the Mesos Framework API)
* src - Contains the entire Mesos source tree. See below for more details about the directories inside of @src@.
* third_party - Contains necessary open source third party software that Mesos leverages for things such as logging, etc.
Modified: incubator/mesos/trunk/docs/Mesos-Roadmap.md
URL: http://svn.apache.org/viewvc/incubator/mesos/trunk/docs/Mesos-Roadmap.md?rev=1452108&r1=1452107&r2=1452108&view=diff
==============================================================================
--- incubator/mesos/trunk/docs/Mesos-Roadmap.md (original)
+++ incubator/mesos/trunk/docs/Mesos-Roadmap.md Sun Mar 3 19:38:25 2013
@@ -5,7 +5,7 @@ The Mesos development roadmap can roughl
1. Mesos Application development
## 1. Growing a development community and promoting adoption
-1. Migrate into Apache Incubator (see the [[incubator proposal|http://wiki.apache.org/incubator/MesosProposal]])
+1. Migrate into Apache Incubator (see the [incubator proposal](http://wiki.apache.org/incubator/MesosProposal))
1. Migrate issues from github issues into JIRA
1. Documentation management and organization
@@ -18,7 +18,7 @@ The Mesos development roadmap can roughl
1. Migrate to ProtocolBuffers as the serialization format (under development at Twitter)
1. User authentication
1. Mesos application debugging support
- 1. More advanced User Interface and [[Event History]] (i.e. logging) - See [[issue #143|https://github.com/mesos/mesos/issuesearch?state=open&q=event#issue/143]] for more details.
+ 1. More advanced User Interface and [Event History](Event-history.md) (i.e. logging) - See [issue #143](https://github.com/mesos/mesos/issuesearch?state=open&q=event#issue/143) for more details.
1. Testing infrastructure, and more tests!
## 3. Mesos applications
@@ -27,7 +27,7 @@ This category of future work is probably
### 3.1 Applications providing cluster OS functionality (e.g. storage, synchronization, naming...)
-The core of Mesos has been designed using the same philosophy behind traditional [[Microkernel operating systems|http://en.wikipedia.org/wiki/Microkernel]]. This means that the core of Mesos (the kernel, if you will) provides a minimal set of low-level abstractions for cluster resources (see [[Mesos Architecture]] for an introduction to the resource offer abstraction). This, in turn means that higher level abstractions (analogous to the filesystem, memory sharing, etc. in traditional operating systems, as well as some abstractions that have no analog in traditional single node operating systems such as DNS).
+The core of Mesos has been designed using the same philosophy behind traditional [Microkernel operating systems](http://en.wikipedia.org/wiki/Microkernel). This means that the core of Mesos (the kernel, if you will) provides a minimal set of low-level abstractions for cluster resources (see [Mesos Architecture](Mesos-Architecture.md) for an introduction to the resource offer abstraction). This, in turn means that higher level abstractions (analogous to the filesystem, memory sharing, etc. in traditional operating systems, as well as some abstractions that have no analog in traditional single node operating systems such as DNS).
those which are primarily intended to be used by other applications (as opposed to being used by human users). They can be seen as operating system modules that implement functionality in the form of services.
Modified: incubator/mesos/trunk/docs/Mesos-configure-command-flag-options.md
URL: http://svn.apache.org/viewvc/incubator/mesos/trunk/docs/Mesos-configure-command-flag-options.md?rev=1452108&r1=1452107&r2=1452108&view=diff
==============================================================================
--- incubator/mesos/trunk/docs/Mesos-configure-command-flag-options.md (original)
+++ incubator/mesos/trunk/docs/Mesos-configure-command-flag-options.md Sun Mar 3 19:38:25 2013
@@ -6,4 +6,4 @@ The configure script itself accepts the
* `--with-webui`: Enable the Mesos web UI (which requires Python 2.6). Recommended.
* `--with-java-home=DIR`: Enable Java application/framework support with a given installation of Java. Required for Hadoop and Spark.
* `--with-java-headers=DIR`: Find Java header files (necessary for newer versions of OS X Snow Leopard).
-* `--with-included-zookeeper` or `--with-zookeeper=DIR`: Enable master fault-tolerance using an existing ZooKeeper installation or the version of ZooKeeper bundled with Mesos. For details, see [[using ZooKeeper]].
+* `--with-included-zookeeper` or `--with-zookeeper=DIR`: Enable master fault-tolerance using an existing ZooKeeper installation or the version of ZooKeeper bundled with Mesos. For details, see [using ZooKeeper](Using-ZooKeeper.textile).
Modified: incubator/mesos/trunk/docs/Mesos-developers-guide.md
URL: http://svn.apache.org/viewvc/incubator/mesos/trunk/docs/Mesos-developers-guide.md?rev=1452108&r1=1452107&r2=1452108&view=diff
==============================================================================
--- incubator/mesos/trunk/docs/Mesos-developers-guide.md (original)
+++ incubator/mesos/trunk/docs/Mesos-developers-guide.md Sun Mar 3 19:38:25 2013
@@ -1,7 +1,7 @@
**This page is up to date as of 2011, Nov 23**
## How to contribute code
-1. Check out the code from the Apache repository either via Git or SVN, get it to build on your machine (instructions on at the wiki [[Home]], we don't currently support building on Windows)
+1. Check out the code from the Apache repository either via Git or SVN, get it to build on your machine (instructions on at the wiki [Home](Home.md), we don't currently support building on Windows)
2. If you encounter a bug while building, report it (see below for instructions)
2. Right now, all active development happens against "trunk" (we aren't yet, as of 2011/11/23, using SVN branches or tags).
@@ -19,7 +19,7 @@
1. Create one or more test cases to exercise the bug or the feature (the Mesos team uses [test-driven development](http://en.wikipedia.org/wiki/Test-driven_development)), before you start coding, make sure these test cases all fail.
1. Make your changes to the code (using whatever IDE/editor you choose) to actually fix the bug or implement the feature.
- 2. Before beginning, please read the [[Mesos C++ Style Guide]]
+ 2. Before beginning, please read the [Mesos C++ Style Guide](Mesos-c++-style-guide.md)
2. Most of your changes will probably be to files inside of <BASE_MESOS_DIR>/src
2. To build, we recommend that you don't build inside of the src directory. We recommend you do the following:
3. From inside of the root Mesos directory: `mkdir build && cd build`
@@ -66,5 +66,4 @@
* A code review request should be created for every JIRA that involves a change to the codebase
## Style Guides
-* [[Mesos C++ Style Guide]]
-* [[Mesos Python Style Guide]]
\ No newline at end of file
+* [Mesos C++ Style Guide](Mesos-c++-style-guide.md)
Modified: incubator/mesos/trunk/docs/Mesos-ready-to-go-AMI.md
URL: http://svn.apache.org/viewvc/incubator/mesos/trunk/docs/Mesos-ready-to-go-AMI.md?rev=1452108&r1=1452107&r2=1452108&view=diff
==============================================================================
--- incubator/mesos/trunk/docs/Mesos-ready-to-go-AMI.md (original)
+++ incubator/mesos/trunk/docs/Mesos-ready-to-go-AMI.md Sun Mar 3 19:38:25 2013
@@ -1,6 +1,6 @@
<font size="3em"><table><tr><td>The most recent functional version of the "ready to go" AMI:</td><td><font size="2em"><b>ami-8a38c8e3</b></font></td></tr></table></font>
-There are two main ways to get a Mesos cluster running on EC2 quickly and easily. One way is via the Mesos [[EC2-Scripts]]. The main guts of the EC2 Scripts are the python program in <mesos-download-root-dir>/src/ec2/mesos_ec2.py which will start up a number of Amazon EC2 Instances (these images already contain a copy of mesos at /root/mesos), and then SSH into those machines, set up the configuration files on the slaves to talk to the master, and also set up HDFS, NFS, and more on those nodes!
+There are two main ways to get a Mesos cluster running on EC2 quickly and easily. One way is via the Mesos [EC2-Scripts](EC2-Scripts.textile). The main guts of the EC2 Scripts are the python program in <mesos-download-root-dir>/src/ec2/mesos_ec2.py which will start up a number of Amazon EC2 Instances (these images already contain a copy of mesos at /root/mesos), and then SSH into those machines, set up the configuration files on the slaves to talk to the master, and also set up HDFS, NFS, and more on those nodes!
The other main way to launch a Mesos cluster on EC2 is using the Mesos "Ready-to-go" AMI. We have set up this AMI to make taking Mesos for a spin as easy as launching a some instances in EC2. That is, we have pre-packaged an AMI with /etc/init.d/mesos-master and /etc/init.d/mesos-slave scripts that make running a Mesos master or slave on an instance of this AMI super easy!
@@ -11,7 +11,7 @@ The other main way to launch a Mesos clu
Here are some high level instructions for getting started with Amazon EC2:
1. Set up you Amazon EC2 user account
-1. Download the [[Amazon EC2 API tools|http://www.amazon.com/gp/redirect.html/ref=aws_rc_ec2tools?location=http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip&token=A80325AA4DAB186C80828ED5138633E3F49160D9]] or install [[Elasticfox|http://aws.amazon.com/developertools/609?_encoding=UTF8&jiveRedirect=1]]
+1. Download the [Amazon EC2 API tools](http://www.amazon.com/gp/redirect.html/ref=aws_rc_ec2tools?location=http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip&token=A80325AA4DAB186C80828ED5138633E3F49160D9) or install [Elasticfox](http://aws.amazon.com/developertools/609?_encoding=UTF8&jiveRedirect=1)
1. Set up your EC2 credentials and try out the tools by running: `ec2-describe-instances`
1. Learn about security groups (which are basically how you set up a firewall on your node) because you'll need to open some ports in order for your Mesos slaves and master to talk to each other (and also to view the webui of the Master)
@@ -25,7 +25,7 @@ Here are some high level instructions fo
## How to use the "ready to go" AMI to run a Mesos Slave node
-1. Run the slave AMI and pass in [[user data|http://docs.amazonwebservices.com/AWSEC2/2007-03-01/DeveloperGuide/AESDG-chapter-instancedata.html]] containing the url of a running Mesos master: `ec2-run-instances <AMI-IDNUM> -d url=1@<master-ip or hostname>:<master port> #see above for most recent AMI-IDNUM to use`
+1. Run the slave AMI and pass in [user data](http://docs.amazonwebservices.com/AWSEC2/2007-03-01/DeveloperGuide/AESDG-chapter-instancedata.html) containing the url of a running Mesos master: `ec2-run-instances <AMI-IDNUM> -d url=1@<master-ip or hostname>:<master port> #see above for most recent AMI-IDNUM to use`
## History of Mesos AMIs
The table below contains more details on the version history of this "ready to go" AMI, which is updated regularly, so check back often!
Modified: incubator/mesos/trunk/docs/Old-mesos-build-instructions.md
URL: http://svn.apache.org/viewvc/incubator/mesos/trunk/docs/Old-mesos-build-instructions.md?rev=1452108&r1=1452107&r2=1452108&view=diff
==============================================================================
--- incubator/mesos/trunk/docs/Old-mesos-build-instructions.md (original)
+++ incubator/mesos/trunk/docs/Old-mesos-build-instructions.md Sun Mar 3 19:38:25 2013
@@ -2,13 +2,13 @@ We migrated the Mesos build system on Ja
<b><font size="4">1) Run one of the configure template scripts</font></b>
-**NOTE:** do not simply run `./configure` without arguments. If you do, your build will fail due to a known issue (see [[MESOS-103|https://issues.apache.org/jira/browse/MESOS-103]] for more details).
+**NOTE:** do not simply run `./configure` without arguments. If you do, your build will fail due to a known issue (see [MESOS-103](https://issues.apache.org/jira/browse/MESOS-103) for more details).
We recommend you use one of the configure.template scripts in the root directory, which will call the more general `configure` script and pass it appropriate arguments. E.g. if you are using OS X, run ./configure.template.macosx.
These configure template scripts contain guesses for the Java and Python paths for distribution indicated in their names (e.g. Mac OS X and several Linux distributions). They assume that you have already installed the packages (i.e. `python-dev` and a JDK). You should double check the configure template script you use (they are just shell scripts, i.e. text files) to make sure the paths it is using for Python and Java match what you have installed. for example, make sure that if you have installed Sun's Java 1.6, your configure template script is not setting JAVA_HOME to be for openjdk.
-Advanced users may wish to run `./configure` directly with their own combination of flag options (see [[Mesos Configure Command Flag Options]]).
+Advanced users may wish to run `./configure` directly with their own combination of flag options (see [Mesos Command-Line Flags](Configuration.textile).
<b><font size="4">2) Run `make`</font></b>
Modified: incubator/mesos/trunk/docs/Running-Hadoop-on-Mesos.md
URL: http://svn.apache.org/viewvc/incubator/mesos/trunk/docs/Running-Hadoop-on-Mesos.md?rev=1452108&r1=1452107&r2=1452108&view=diff
==============================================================================
--- incubator/mesos/trunk/docs/Running-Hadoop-on-Mesos.md (original)
+++ incubator/mesos/trunk/docs/Running-Hadoop-on-Mesos.md Sun Mar 3 19:38:25 2013
@@ -5,7 +5,7 @@ You can build the ported version of Hado
To run Hadoop on Mesos, follow these steps:
<ol>
<li> Run `make hadoop` to build Hadoop 0.20.205.0 with Mesos support, or `TUTORIAL.sh` to patch and build your own Hadoop version.</li>
-<li> Set up [[Hadoop's configuration|http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/]] as you would usually do with a new install of Hadoop, following the [[instructions on the Hadoop website|http://hadoop.apache.org/common/docs/r0.20.2/index.html]] (at the very least, you need to set <code>JAVA_HOME</code> in Hadoop's <code>conf/hadoop-env.sh</code> and set <code>mapred.job.tracker</code> in <code>conf/mapred-site.xml</code>).</li>
+<li> Set up [Hadoop's configuration](http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster) as you would usually do with a new install of Hadoop, following the [instructions on the Hadoop website](http://hadoop.apache.org/common/docs/r0.20.2/index.html) (at the very least, you need to set <code>JAVA_HOME</code> in Hadoop's <code>conf/hadoop-env.sh</code> and set <code>mapred.job.tracker</code> in <code>conf/mapred-site.xml</code>).</li>
</li>
<li> Add the following parameters to Hadoop's <code>conf/mapred-site.xml</code>:
<pre>
Modified: incubator/mesos/trunk/docs/Running-Mesos-On-Mac-OS-X-Snow-Leopard-(Single-Node-Cluster).md
URL: http://svn.apache.org/viewvc/incubator/mesos/trunk/docs/Running-Mesos-On-Mac-OS-X-Snow-Leopard-%28Single-Node-Cluster%29.md?rev=1452108&r1=1452107&r2=1452108&view=diff
==============================================================================
--- incubator/mesos/trunk/docs/Running-Mesos-On-Mac-OS-X-Snow-Leopard-(Single-Node-Cluster).md (original)
+++ incubator/mesos/trunk/docs/Running-Mesos-On-Mac-OS-X-Snow-Leopard-(Single-Node-Cluster).md Sun Mar 3 19:38:25 2013
@@ -298,5 +298,5 @@ drwxr-xr-x - billz supergroup
* mesos-dev@incubator.apache.org
## Want to contribute?
-* Check out the [[Mesos Developers Guide]]
+* Check out the [Mesos Developers Guide](Mesos-developers-guide.md)
* [Submit a Bug](https://issues.apache.org/jira/browse/MESOS)
\ No newline at end of file
Modified: incubator/mesos/trunk/docs/Running-a-web-application-farm-on-mesos.textile
URL: http://svn.apache.org/viewvc/incubator/mesos/trunk/docs/Running-a-web-application-farm-on-mesos.textile?rev=1452108&r1=1452107&r2=1452108&view=diff
==============================================================================
--- incubator/mesos/trunk/docs/Running-a-web-application-farm-on-mesos.textile (original)
+++ incubator/mesos/trunk/docs/Running-a-web-application-farm-on-mesos.textile Sun Mar 3 19:38:25 2013
@@ -1,10 +1,10 @@
h1. Running a web-farm on Mesos in your private cluster
-Mesos comes with a framework written in Python that will run a web load balancer ([[HAProxy|http://haproxy.1wt.eu/]]) and use apache for the backend servers. The framework is in MESOS_HOME/frameworks/haproxy+apache. There is a README in that directory that might be slightly helpful, but we recommend looking a the framework scheduler and executor code to familiarize yourself with the concepts behind the framework.
+Mesos comes with a framework written in Python that will run a web load balancer ("HAProxy":http://haproxy.1wt.eu) and use apache for the backend servers. The framework is in MESOS_HOME/frameworks/haproxy+apache. There is a README in that directory that might be slightly helpful, but we recommend looking a the framework scheduler and executor code to familiarize yourself with the concepts behind the framework.
-HAProxy has this annoying artifact that in order to ramp up and down in the number of servers it is load balancing between it has to kill the existing running HAProxy process and start a new one, which inevitably leads to dropped connections. Some people have looked into ways to get around the dropped connections (see this animoto blog post about adding/removing servers to haproxy: [[http://labs.animoto.com/2010/03/10/uptime-with-haproxy/]]).
+HAProxy has this annoying artifact that in order to ramp up and down in the number of servers it is load balancing between it has to kill the existing running HAProxy process and start a new one, which inevitably leads to dropped connections. Some people have looked into ways to get around the dropped connections (see this animoto blog post about adding/removing servers to haproxy: http://labs.animoto.com/2010/03/10/uptime-with-haproxy).
-We recommend checking out other load balancers as well. Basically, whatever load balancer you're using, the framework scheduler should query the statistics of the load balancer to figure out how many web requests are being served to it and then ramp up and down the number of web server tasks based on those statistics. We've also written a framework that uses [[Linux Virtual Server (LVS)|http://www.linux-vs.org/]] as the load balancer. LVS can be run in three different modes, and any should work with Mesos.
+We recommend checking out other load balancers as well. Basically, whatever load balancer you're using, the framework scheduler should query the statistics of the load balancer to figure out how many web requests are being served to it and then ramp up and down the number of web server tasks based on those statistics. We've also written a framework that uses "Linux Virtual Server (LVS)":http://www.linux-vs.org as the load balancer. LVS can be run in three different modes, and any should work with Mesos.
h1. Running a web farm on Mesos on EC2
@@ -16,6 +16,6 @@ We have also found that we can allow mul
h2. Where to look to get started
-We have also successfully used Amazon's Elastic Load Balancer instead of HAProxy. This is still in a non-master development branch called [[andyk-elb+apache-notorque|https://github.com/mesos/mesos/tree/andyk-elb+apache-notorque]]. You will need to set up a Mesos EC2 cluster and the framework uses python boto to connect to Amazon's CloudWatch and EC2 so you will need to create a credentials file for boto to be able to connect to those amazon services correctly from the node on which you run the framework.
+We have also successfully used Amazon's Elastic Load Balancer instead of HAProxy. This is still in a non-master development branch called "andyk-elb+apache-notorque":https://github.com/mesos/mesos/tree/andyk-elb+apache-notorque. You will need to set up a Mesos EC2 cluster and the framework uses python boto to connect to Amazon's CloudWatch and EC2 so you will need to create a credentials file for boto to be able to connect to those amazon services correctly from the node on which you run the framework.
Watch out when using ELB however, since it displays some interesting and non-intuitive behavior when ramping up and down (though it doesn't drop connections like HAProxy does out of the box). See this blog post: http://shlomoswidler.com/2009/07/elastic-in-elastic-load-balancing-elb.html for details about how ELB ramps up and down and especially read it if you plan on.
Modified: incubator/mesos/trunk/docs/Using-Linux-Containers.textile
URL: http://svn.apache.org/viewvc/incubator/mesos/trunk/docs/Using-Linux-Containers.textile?rev=1452108&r1=1452107&r2=1452108&view=diff
==============================================================================
--- incubator/mesos/trunk/docs/Using-Linux-Containers.textile (original)
+++ incubator/mesos/trunk/docs/Using-Linux-Containers.textile Sun Mar 3 19:38:25 2013
@@ -4,7 +4,7 @@ Mesos can use Linux Containers to isolat
* Install the Linux container userspace tools (e.g., on Ubuntu, @apt-get install lxc@) and ensure that you are running a version of the kernel with containers enabled using @lxc-checkconfig@. In particular, for memory isolation, this should report that the memory subsystem is enabled.
* Mount the Linux container virtual filesystem, for example by creating the directory @/cgroup@ and using @mount -t cgroup cgroup /cgroup@. ("This guide":http://lxc.teegra.net/ is a good tutorial on Linux containers).
* Make sure that you are running @mesos-slave@ as root so that it can use the container tools.
-* Pass the argument @--isolation=lxc@ to @mesos-slave@ (or set @isolation@ to @lxc@ in the [[config file|Configuration]]).
+* Pass the argument @--isolation=lxc@ to @mesos-slave@.
The Linux container isolation module will perform some sanity checks upon launching to make sure that it can use containers, and will then attempt to use them for frameworks. The containers will be named @mesos.slave-X.framework-Y@, where X and Y are the slave and framework IDs.
Added: incubator/mesos/trunk/docs/images/architecture-example.jpg
URL: http://svn.apache.org/viewvc/incubator/mesos/trunk/docs/images/architecture-example.jpg?rev=1452108&view=auto
==============================================================================
Files incubator/mesos/trunk/docs/images/architecture-example.jpg (added) and incubator/mesos/trunk/docs/images/architecture-example.jpg Sun Mar 3 19:38:25 2013 differ
Added: incubator/mesos/trunk/docs/images/architecture3.jpg
URL: http://svn.apache.org/viewvc/incubator/mesos/trunk/docs/images/architecture3.jpg?rev=1452108&view=auto
==============================================================================
Files incubator/mesos/trunk/docs/images/architecture3.jpg (added) and incubator/mesos/trunk/docs/images/architecture3.jpg Sun Mar 3 19:38:25 2013 differ