You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mesos.apache.org by dl...@apache.org on 2013/10/28 06:27:43 UTC
[2/2] git commit: MESOS-511 Update documentation home and remove
outdated drafts pages.
MESOS-511 Update documentation home and remove outdated drafts pages.
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/5cd623e3
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/5cd623e3
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/5cd623e3
Branch: refs/heads/master
Commit: 5cd623e38d86648522b038115c36701c579cc8a7
Parents: 5223215
Author: Dave Lester <dl...@twitter.com>
Authored: Sun Oct 27 22:27:12 2013 -0700
Committer: Dave Lester <dl...@twitter.com>
Committed: Sun Oct 27 22:27:12 2013 -0700
----------------------------------------------------------------------
docs/Allocation-module.md | 132 ++++++++
docs/Allocation-module.textile | 128 --------
docs/App-Framework-Development-Guide.md | 208 +++++++++++++
docs/App-Framework-development-guide.textile | 202 -------------
docs/Configuration.md | 131 ++++----
docs/Deploy-Scripts.md | 16 +
docs/Deploy-Scripts.textile | 9 -
docs/EC2-Scripts.md | 59 ++++
docs/EC2-Scripts.textile | 52 ----
docs/Event-history.md | 3 -
docs/Getting-Started.md | 21 +-
docs/Hadoop-demo-VM.md | 79 -----
docs/High-Availability.md | 43 +--
docs/Home.md | 131 ++------
docs/Logging-and-Debugging.md | 11 +
docs/Logging-and-Debugging.textile | 5 -
docs/Mesos-Architecture.md | 26 +-
docs/Mesos-Code-Internals.md | 41 +++
docs/Mesos-Code-Internals.textile | 32 --
docs/Mesos-Presentations.md | 24 +-
docs/Mesos-Roadmap.md | 6 +
docs/Mesos-c++-style-guide.md | 32 +-
docs/Mesos-configure-command-flag-options.md | 9 -
docs/Mesos-developers-guide.md | 49 +--
docs/Mesos-ready-to-go-AMI.md | 65 ----
docs/Old-mesos-build-instructions.md | 16 -
docs/Powered-by-Mesos.md | 11 +-
...c-OS-X-Snow-Leopard-(Single-Node-Cluster).md | 302 -------------------
...sos-on-Ubuntu-Linux-(Single-Node-Cluster).md | 284 -----------------
docs/Running-a-web-application-farm-on-mesos.md | 25 ++
...ning-a-web-application-farm-on-mesos.textile | 21 --
docs/Running-torque-or-mpi-on-mesos.md | 8 +-
docs/Slave-Recovery.md | 53 ++--
docs/Upgrades.md | 66 ++--
docs/Using-Linux-Containers.textile | 19 --
docs/Using-the-mesos-submit-tool.md | 9 +-
36 files changed, 791 insertions(+), 1537 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/mesos/blob/5cd623e3/docs/Allocation-module.md
----------------------------------------------------------------------
diff --git a/docs/Allocation-module.md b/docs/Allocation-module.md
new file mode 100644
index 0000000..b308a23
--- /dev/null
+++ b/docs/Allocation-module.md
@@ -0,0 +1,132 @@
+---
+layout: documentation
+---
+
+# Mesos Allocation Module
+
+The logic that the Mesos master uses to determine which frameworks to make offer resource offers to is encapsulated in the Master's _allocation module_. The allocation module is a pluggable component that organizations can use to implement their own sharing policy, e.g. fair-sharing, Dominant Resource Fairness (see [the DRF paper](http://www.eecs.berkeley.edu/Pubs/TechRpts/2010/EECS-2010-55.pdf)), priority, etc.
+
+## Allocation Module API
+
+Mesos is implemented in C++, so allocation modules are implemented in C++, and inherit the @AllocatorProcess@ class defined in @MESOS_HOME/src/master/allocator.hpp@. As of the time of this writing (5/29/13), the API for allocation modules is as follows:
+
+```
+ virtual ~AllocatorProcess() {}
+
+ virtual void initialize(
+ const Flags& flags,
+ const process::PID<Master>& master,
+ const hashmap<std::string, RoleInfo>& roles) = 0;
+
+ virtual void frameworkAdded(
+ const FrameworkID& frameworkId,
+ const FrameworkInfo& frameworkInfo,
+ const Resources& used) = 0;
+
+ virtual void frameworkRemoved(
+ const FrameworkID& frameworkId) = 0;
+
+ virtual void frameworkActivated(
+ const FrameworkID& frameworkId,
+ const FrameworkInfo& frameworkInfo) = 0;
+
+ virtual void frameworkDeactivated(
+ const FrameworkID& frameworkId) = 0;
+
+ virtual void slaveAdded(
+ const SlaveID& slaveId,
+ const SlaveInfo& slaveInfo,
+ const hashmap<FrameworkID, Resources>& used) = 0;
+
+ virtual void slaveRemoved(
+ const SlaveID& slaveId) = 0;
+
+ virtual void updateWhitelist(
+ const Option<hashset<std::string> >& whitelist) = 0;
+
+ virtual void resourcesRequested(
+ const FrameworkID& frameworkId,
+ const std::vector<Request>& requests) = 0;
+
+ // Whenever resources offered to a framework go unused (e.g.,
+ // refused) the master invokes this callback.
+ virtual void resourcesUnused(
+ const FrameworkID& frameworkId,
+ const SlaveID& slaveId,
+ const Resources& resources,
+ const Option<Filters>& filters) = 0;
+
+ // Whenever resources are "recovered" in the cluster (e.g., a task
+ // finishes, an offer is removed because a framework has failed or
+ // is failing over) the master invokes this callback.
+ virtual void resourcesRecovered(
+ const FrameworkID& frameworkId,
+ const SlaveID& slaveId,
+ const Resources& resources) = 0;
+
+ // Whenever a framework that has filtered resources wants to revive
+ // offers for those resources the master invokes this callback.
+ virtual void offersRevived(
+ const FrameworkID& frameworkId) = 0;
+```
+
+The default allocation module is the HierarchicalAllocatorProcess, which can be found in @MESOS_HOME/src/master/hierarchical_allocator_process.hpp@. You can reference this as a starting place if you choose to write your own allocation module.
+
+## Sorter API
+
+Additionally, the hierarchical allocator module can be extended without the need to reimplement the entirety of the allocation logic through the use of the @Sorter@ abstraction.
+
+Sorters define the order that roles or frameworks should be offered resources in by taking "client" objects and some information about those clients and returning an ordered list of clients.
+
+Sorters are implemented in C++ and inherit the @Sorter@ class defined in @MESOS_HOME/src/master/sorter.hpp@. As of the time of this writing, the API for Sorters is as follows:
+
+```
+ virtual ~Sorter() {}
+
+ // Adds a client to allocate resources to. A client
+ // may be a user or a framework.
+ virtual void add(const std::string& client, double weight = 1) = 0;
+
+ // Removes a client.
+ virtual void remove(const std::string& client) = 0;
+
+ // Readds a client to the sort after deactivate.
+ virtual void activate(const std::string& client) = 0;
+
+ // Removes a client from the sort, so it won't get allocated to.
+ virtual void deactivate(const std::string& client) = 0;
+
+ // Specify that resources have been allocated to the given client.
+ virtual void allocated(const std::string& client,
+ const Resources& resources) = 0;
+
+ // Specify that resources have been unallocated from the given client.
+ virtual void unallocated(const std::string& client,
+ const Resources& resources) = 0;
+
+ // Returns the resources that have been allocated to this client.
+ virtual Resources allocation(const std::string& client) = 0;
+
+ // Add resources to the total pool of resources this
+ // Sorter should consider.
+ virtual void add(const Resources& resources) = 0;
+
+ // Remove resources from the total pool.
+ virtual void remove(const Resources& resources) = 0;
+
+ // Returns a list of all clients, in the order that they
+ // should be allocated to, according to this Sorter's policy.
+ virtual std::list<std::string> sort() = 0;
+
+ // Returns true if this Sorter contains the specified client,
+ // either active or deactivated.
+ virtual bool contains(const std::string& client) = 0;
+
+ // Returns the number of clients this Sorter contains,
+ // either active or deactivated.
+ virtual int count() = 0;
+```
+
+The default @Sorter@ is the DRFSorter, which implements fair sharing and can be found at @MESOS_HOME/src/master/drf_sorter.hpp@.
+
+For DRF, if weights are specified in Sorter::add, a client's share will be divided by the weight, creating a form of priority. For example, a role that has a weight of 2 will be offered twice as many resources as a role with weight 1.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/mesos/blob/5cd623e3/docs/Allocation-module.textile
----------------------------------------------------------------------
diff --git a/docs/Allocation-module.textile b/docs/Allocation-module.textile
deleted file mode 100644
index c85bb6e..0000000
--- a/docs/Allocation-module.textile
+++ /dev/null
@@ -1,128 +0,0 @@
-h1. Overview
-
-The logic that the Mesos master uses to determine which frameworks to make offer resource offers to is encapsulated in the Master's _allocation module_. The allocation module is a pluggable component that organizations can use to implement their own sharing policy, e.g. fair-sharing, Dominant Resource Fairness (see "the DRF paper":http://www.eecs.berkeley.edu/Pubs/TechRpts/2010/EECS-2010-55.pdf), priority, etc.
-
-h1. Allocation Module API
-
-Mesos is implemented in C++, so allocation modules are implemented in C++, and inherit the @AllocatorProcess@ class defined in @MESOS_HOME/src/master/allocator.hpp@. As of the time of this writing (5/29/13), the API for allocation modules is as follows:
-
-<pre>
- virtual ~AllocatorProcess() {}
-
- virtual void initialize(
- const Flags& flags,
- const process::PID<Master>& master,
- const hashmap<std::string, RoleInfo>& roles) = 0;
-
- virtual void frameworkAdded(
- const FrameworkID& frameworkId,
- const FrameworkInfo& frameworkInfo,
- const Resources& used) = 0;
-
- virtual void frameworkRemoved(
- const FrameworkID& frameworkId) = 0;
-
- virtual void frameworkActivated(
- const FrameworkID& frameworkId,
- const FrameworkInfo& frameworkInfo) = 0;
-
- virtual void frameworkDeactivated(
- const FrameworkID& frameworkId) = 0;
-
- virtual void slaveAdded(
- const SlaveID& slaveId,
- const SlaveInfo& slaveInfo,
- const hashmap<FrameworkID, Resources>& used) = 0;
-
- virtual void slaveRemoved(
- const SlaveID& slaveId) = 0;
-
- virtual void updateWhitelist(
- const Option<hashset<std::string> >& whitelist) = 0;
-
- virtual void resourcesRequested(
- const FrameworkID& frameworkId,
- const std::vector<Request>& requests) = 0;
-
- // Whenever resources offered to a framework go unused (e.g.,
- // refused) the master invokes this callback.
- virtual void resourcesUnused(
- const FrameworkID& frameworkId,
- const SlaveID& slaveId,
- const Resources& resources,
- const Option<Filters>& filters) = 0;
-
- // Whenever resources are "recovered" in the cluster (e.g., a task
- // finishes, an offer is removed because a framework has failed or
- // is failing over) the master invokes this callback.
- virtual void resourcesRecovered(
- const FrameworkID& frameworkId,
- const SlaveID& slaveId,
- const Resources& resources) = 0;
-
- // Whenever a framework that has filtered resources wants to revive
- // offers for those resources the master invokes this callback.
- virtual void offersRevived(
- const FrameworkID& frameworkId) = 0;
-</pre>
-
-The default allocation module is the HierarchicalAllocatorProcess, which can be found in @MESOS_HOME/src/master/hierarchical_allocator_process.hpp@. You can reference this as a starting place if you choose to write your own allocation module.
-
-h1. Sorter API
-
-Additionally, the hierarchical allocator module can be extended without the need to reimplement the entirety of the allocation logic through the use of the @Sorter@ abstraction.
-
-Sorters define the order that roles or frameworks should be offered resources in by taking "client" objects and some information about those clients and returning an ordered list of clients.
-
-Sorters are implemented in C++ and inherit the @Sorter@ class defined in @MESOS_HOME/src/master/sorter.hpp@. As of the time of this writing, the API for Sorters is as follows:
-
-<pre>
- virtual ~Sorter() {}
-
- // Adds a client to allocate resources to. A client
- // may be a user or a framework.
- virtual void add(const std::string& client, double weight = 1) = 0;
-
- // Removes a client.
- virtual void remove(const std::string& client) = 0;
-
- // Readds a client to the sort after deactivate.
- virtual void activate(const std::string& client) = 0;
-
- // Removes a client from the sort, so it won't get allocated to.
- virtual void deactivate(const std::string& client) = 0;
-
- // Specify that resources have been allocated to the given client.
- virtual void allocated(const std::string& client,
- const Resources& resources) = 0;
-
- // Specify that resources have been unallocated from the given client.
- virtual void unallocated(const std::string& client,
- const Resources& resources) = 0;
-
- // Returns the resources that have been allocated to this client.
- virtual Resources allocation(const std::string& client) = 0;
-
- // Add resources to the total pool of resources this
- // Sorter should consider.
- virtual void add(const Resources& resources) = 0;
-
- // Remove resources from the total pool.
- virtual void remove(const Resources& resources) = 0;
-
- // Returns a list of all clients, in the order that they
- // should be allocated to, according to this Sorter's policy.
- virtual std::list<std::string> sort() = 0;
-
- // Returns true if this Sorter contains the specified client,
- // either active or deactivated.
- virtual bool contains(const std::string& client) = 0;
-
- // Returns the number of clients this Sorter contains,
- // either active or deactivated.
- virtual int count() = 0;
-</pre>
-
-The default @Sorter@ is the DRFSorter, which implements fair sharing and can be found at @MESOS_HOME/src/master/drf_sorter.hpp@.
-
-For DRF, if weights are specified in Sorter::add, a client's share will be divided by the weight, creating a form of priority. For example, a role that has a weight of 2 will be offered twice as many resources as a role with weight 1.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/mesos/blob/5cd623e3/docs/App-Framework-Development-Guide.md
----------------------------------------------------------------------
diff --git a/docs/App-Framework-Development-Guide.md b/docs/App-Framework-Development-Guide.md
new file mode 100644
index 0000000..6d75e4a
--- /dev/null
+++ b/docs/App-Framework-Development-Guide.md
@@ -0,0 +1,208 @@
+---
+layout: documentation
+---
+
+# Framework Development Guide
+
+In this document we refer to Mesos applications as "frameworks".
+
+See one of the example framework schedulers in @MESOS_HOME/src/examples/@ to get an idea of what a Mesos framework scheduler and executor in the language of your choice looks like.
+
+## Create your Framework Scheduler
+
+You can write a framework scheduler in C, C++, Java/Scala, or Python. Your framework scheduler should inherit from the @Scheduler@ class (see API below). Your scheduler should create a SchedulerDriver (which will mediate communication between your scheduler and the Mesos master) and then call @SchedulerDriver.run()@
+
+### Scheduler API
+
+Declared in @MESOS_HOME/include/mesos/scheduler.hpp@
+
+```
+ /**
+ * Empty virtual destructor (necessary to instantiate subclasses).
+ */
+ virtual ~Scheduler() {}
+
+ /**
+ * Invoked when the scheduler successfully registers with a Mesos
+ * master. A unique ID (generated by the master) used for
+ * distinguishing this framework from others and MasterInfo
+ * with the ip and port of the current master are provided as arguments.
+ */
+ virtual void registered(SchedulerDriver* driver,
+ const FrameworkID& frameworkId,
+ const MasterInfo& masterInfo) = 0;
+
+ /**
+ * Invoked when the scheduler re-registers with a newly elected Mesos master.
+ * This is only called when the scheduler has previously been registered.
+ * MasterInfo containing the updated information about the elected master
+ * is provided as an argument.
+ */
+ virtual void reregistered(SchedulerDriver* driver,
+ const MasterInfo& masterInfo) = 0;
+
+ /**
+ * Invoked when the scheduler becomes "disconnected" from the master
+ * (e.g., the master fails and another is taking over).
+ */
+ virtual void disconnected(SchedulerDriver* driver) = 0;
+
+ /**
+ * Invoked when resources have been offered to this framework. A
+ * single offer will only contain resources from a single slave.
+ * Resources associated with an offer will not be re-offered to
+ * _this_ framework until either (a) this framework has rejected
+ * those resources (see SchedulerDriver::launchTasks) or (b) those
+ * resources have been rescinded (see Scheduler::offerRescinded).
+ * Note that resources may be concurrently offered to more than one
+ * framework at a time (depending on the allocator being used). In
+ * that case, the first framework to launch tasks using those
+ * resources will be able to use them while the other frameworks
+ * will have those resources rescinded (or if a framework has
+ * already launched tasks with those resources then those tasks will
+ * fail with a TASK_LOST status and a message saying as much).
+ */
+ virtual void resourceOffers(SchedulerDriver* driver,
+ const std::vector<Offer>& offers) = 0;
+
+ /**
+ * Invoked when an offer is no longer valid (e.g., the slave was
+ * lost or another framework used resources in the offer). If for
+ * whatever reason an offer is never rescinded (e.g., dropped
+ * message, failing over framework, etc.), a framwork that attempts
+ * to launch tasks using an invalid offer will receive TASK_LOST
+ * status updats for those tasks (see Scheduler::resourceOffers).
+ */
+ virtual void offerRescinded(SchedulerDriver* driver,
+ const OfferID& offerId) = 0;
+
+ /**
+ * Invoked when the status of a task has changed (e.g., a slave is
+ * lost and so the task is lost, a task finishes and an executor
+ * sends a status update saying so, etc). Note that returning from
+ * this callback _acknowledges_ receipt of this status update! If
+ * for whatever reason the scheduler aborts during this callback (or
+ * the process exits) another status update will be delivered (note,
+ * however, that this is currently not true if the slave sending the
+ * status update is lost/fails during that time).
+ */
+ virtual void statusUpdate(SchedulerDriver* driver,
+ const TaskStatus& status) = 0;
+
+ /**
+ * Invoked when an executor sends a message. These messages are best
+ * effort; do not expect a framework message to be retransmitted in
+ * any reliable fashion.
+ */
+ virtual void frameworkMessage(SchedulerDriver* driver,
+ const ExecutorID& executorId,
+ const SlaveID& slaveId,
+ const std::string& data) = 0;
+
+ /**
+ * Invoked when a slave has been determined unreachable (e.g.,
+ * machine failure, network partition). Most frameworks will need to
+ * reschedule any tasks launched on this slave on a new slave.
+ */
+ virtual void slaveLost(SchedulerDriver* driver,
+ const SlaveID& slaveId) = 0;
+
+ /**
+ * Invoked when an executor has exited/terminated. Note that any
+ * tasks running will have TASK_LOST status updates automagically
+ * generated.
+ */
+ virtual void executorLost(SchedulerDriver* driver,
+ const ExecutorID& executorId,
+ const SlaveID& slaveId,
+ int status) = 0;
+
+ /**
+ * Invoked when there is an unrecoverable error in the scheduler or
+ * scheduler driver. The driver will be aborted BEFORE invoking this
+ * callback.
+ */
+ virtual void error(SchedulerDriver* driver, const std::string& message) = 0;
+```
+
+## Create your Framework Executor
+
+Your framework executor must inherit from the Executor class. It must override the launchTask() method. You can use the $MESOS_HOME environment variable inside of your executor to determine where mesos is running from.
+
+### Executor API
+
+Declared in @MESOS_HOME/include/mesos/executor.hpp@
+
+```
+ /**
+ * Invoked once the executor driver has been able to successfully
+ * connect with Mesos. In particular, a scheduler can pass some
+ * data to it's executors through the FrameworkInfo.ExecutorInfo's
+ * data field.
+ */
+ virtual void registered(ExecutorDriver* driver,
+ const ExecutorInfo& executorInfo,
+ const FrameworkInfo& frameworkInfo,
+ const SlaveInfo& slaveInfo) = 0;
+
+ /**
+ * Invoked when the executor re-registers with a restarted slave.
+ */
+ virtual void reregistered(ExecutorDriver* driver,
+ const SlaveInfo& slaveInfo) = 0;
+
+ /**
+ * Invoked when the executor becomes "disconnected" from the slave
+ * (e.g., the slave is being restarted due to an upgrade).
+ */
+ virtual void disconnected(ExecutorDriver* driver) = 0;
+
+ /**
+ * Invoked when a task has been launched on this executor (initiated
+ * via Scheduler::launchTasks). Note that this task can be realized
+ * with a thread, a process, or some simple computation, however, no
+ * other callbacks will be invoked on this executor until this
+ * callback has returned.
+ */
+ virtual void launchTask(ExecutorDriver* driver,
+ const TaskInfo& task) = 0;
+
+ /**
+ * Invoked when a task running within this executor has been killed
+ * (via SchedulerDriver::killTask). Note that no status update will
+ * be sent on behalf of the executor, the executor is responsible
+ * for creating a new TaskStatus (i.e., with TASK_KILLED) and
+ * invoking ExecutorDriver::sendStatusUpdate.
+ */
+ virtual void killTask(ExecutorDriver* driver, const TaskID& taskId) = 0;
+
+ /**
+ * Invoked when a framework message has arrived for this
+ * executor. These messages are best effort; do not expect a
+ * framework message to be retransmitted in any reliable fashion.
+ */
+ virtual void frameworkMessage(ExecutorDriver* driver,
+ const std::string& data) = 0;
+
+ /**
+ * Invoked when the executor should terminate all of it's currently
+ * running tasks. Note that after a Mesos has determined that an
+ * executor has terminated any tasks that the executor did not send
+ * terminal status updates for (e.g., TASK_KILLED, TASK_FINISHED,
+ * TASK_FAILED, etc) a TASK_LOST status update will be created.
+ */
+ virtual void shutdown(ExecutorDriver* driver) = 0;
+
+ /**
+ * Invoked when a fatal error has occured with the executor and/or
+ * executor driver. The driver will be aborted BEFORE invoking this
+ * callback.
+ */
+ virtual void error(ExecutorDriver* driver, const std::string& message) = 0;
+```
+
+## Install your Framework
+
+You need to put your framework somewhere that all slaves on the cluster can get it from. If you are running HDFS, you can put your executor into HDFS. Then, you tell Mesos where it is via the @ExecutorInfo@ parameter of @MesosSchedulerDriver@'s constructor (e.g. see src/examples/java/TestFramework.java for an example of this). ExecutorInfo is a a Protocol Buffer Message class (defined in @include/mesos/mesos.proto@), and you set its uri field to something like "HDFS://path/to/executor/". Also, you can pass the @frameworks_home@ configuration option (defaults to: @MESOS_HOME/frameworks@) to your @mesos-slave@ daemons when you launch them to specify where all of your framework executors are stored (e.g. on an NFS mount that is available to all slaves), then set @ExecutorInfo@ to be a relative path, and the slave will prepend the value of frameworks_home to the relative path provided.
+
+Once you are sure that your executors are available to the mesos-slaves, you should be able to run your scheduler, which will register with the Mesos master, and start receiving resource offers!
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/mesos/blob/5cd623e3/docs/App-Framework-development-guide.textile
----------------------------------------------------------------------
diff --git a/docs/App-Framework-development-guide.textile b/docs/App-Framework-development-guide.textile
deleted file mode 100644
index ebda9a5..0000000
--- a/docs/App-Framework-development-guide.textile
+++ /dev/null
@@ -1,202 +0,0 @@
-h1. App/Framework development guide
-
-<b>Note:</b> In this document (and also in the Mesos code base, at least as of Feb 14, 2012), we refer to Mesos Applications as "Frameworks".
-
-See one of the example framework schedulers in @MESOS_HOME/src/examples/@ to get an idea of what a Mesos framework scheduler and executor in the language of your choice looks like.
-
-h2. Create your Framework Scheduler
-
-You can write a framework scheduler in C, C++, Java/Scala, or Python. Your framework scheduler should inherit from the @Scheduler@ class (see API below). Your scheduler should create a SchedulerDriver (which will mediate communication between your scheduler and the Mesos master) and then call @SchedulerDriver.run()@
-
-h3. Scheduler API (as of 2013/02/01) declared in @MESOS_HOME/include/mesos/scheduler.hpp@
-
-```c++
- /**
- * Empty virtual destructor (necessary to instantiate subclasses).
- */
- virtual ~Scheduler() {}
-
- /**
- * Invoked when the scheduler successfully registers with a Mesos
- * master. A unique ID (generated by the master) used for
- * distinguishing this framework from others and MasterInfo
- * with the ip and port of the current master are provided as arguments.
- */
- virtual void registered(SchedulerDriver* driver,
- const FrameworkID& frameworkId,
- const MasterInfo& masterInfo) = 0;
-
- /**
- * Invoked when the scheduler re-registers with a newly elected Mesos master.
- * This is only called when the scheduler has previously been registered.
- * MasterInfo containing the updated information about the elected master
- * is provided as an argument.
- */
- virtual void reregistered(SchedulerDriver* driver,
- const MasterInfo& masterInfo) = 0;
-
- /**
- * Invoked when the scheduler becomes "disconnected" from the master
- * (e.g., the master fails and another is taking over).
- */
- virtual void disconnected(SchedulerDriver* driver) = 0;
-
- /**
- * Invoked when resources have been offered to this framework. A
- * single offer will only contain resources from a single slave.
- * Resources associated with an offer will not be re-offered to
- * _this_ framework until either (a) this framework has rejected
- * those resources (see SchedulerDriver::launchTasks) or (b) those
- * resources have been rescinded (see Scheduler::offerRescinded).
- * Note that resources may be concurrently offered to more than one
- * framework at a time (depending on the allocator being used). In
- * that case, the first framework to launch tasks using those
- * resources will be able to use them while the other frameworks
- * will have those resources rescinded (or if a framework has
- * already launched tasks with those resources then those tasks will
- * fail with a TASK_LOST status and a message saying as much).
- */
- virtual void resourceOffers(SchedulerDriver* driver,
- const std::vector<Offer>& offers) = 0;
-
- /**
- * Invoked when an offer is no longer valid (e.g., the slave was
- * lost or another framework used resources in the offer). If for
- * whatever reason an offer is never rescinded (e.g., dropped
- * message, failing over framework, etc.), a framwork that attempts
- * to launch tasks using an invalid offer will receive TASK_LOST
- * status updats for those tasks (see Scheduler::resourceOffers).
- */
- virtual void offerRescinded(SchedulerDriver* driver,
- const OfferID& offerId) = 0;
-
- /**
- * Invoked when the status of a task has changed (e.g., a slave is
- * lost and so the task is lost, a task finishes and an executor
- * sends a status update saying so, etc). Note that returning from
- * this callback _acknowledges_ receipt of this status update! If
- * for whatever reason the scheduler aborts during this callback (or
- * the process exits) another status update will be delivered (note,
- * however, that this is currently not true if the slave sending the
- * status update is lost/fails during that time).
- */
- virtual void statusUpdate(SchedulerDriver* driver,
- const TaskStatus& status) = 0;
-
- /**
- * Invoked when an executor sends a message. These messages are best
- * effort; do not expect a framework message to be retransmitted in
- * any reliable fashion.
- */
- virtual void frameworkMessage(SchedulerDriver* driver,
- const ExecutorID& executorId,
- const SlaveID& slaveId,
- const std::string& data) = 0;
-
- /**
- * Invoked when a slave has been determined unreachable (e.g.,
- * machine failure, network partition). Most frameworks will need to
- * reschedule any tasks launched on this slave on a new slave.
- */
- virtual void slaveLost(SchedulerDriver* driver,
- const SlaveID& slaveId) = 0;
-
- /**
- * Invoked when an executor has exited/terminated. Note that any
- * tasks running will have TASK_LOST status updates automagically
- * generated.
- */
- virtual void executorLost(SchedulerDriver* driver,
- const ExecutorID& executorId,
- const SlaveID& slaveId,
- int status) = 0;
-
- /**
- * Invoked when there is an unrecoverable error in the scheduler or
- * scheduler driver. The driver will be aborted BEFORE invoking this
- * callback.
- */
- virtual void error(SchedulerDriver* driver, const std::string& message) = 0;
-```
-
-h2. Create your Framework Executor
-
-Your framework executor must inherit from the Executor class. It must override the launchTask() method.
-You can use the $MESOS_HOME environment variable inside of your executor to determine where mesos is running from.
-
-h3. Executor API (as of 2013/02/01) declared in @MESOS_HOME/include/mesos/executor.hpp@
-
-```c++
- /**
- * Invoked once the executor driver has been able to successfully
- * connect with Mesos. In particular, a scheduler can pass some
- * data to it's executors through the FrameworkInfo.ExecutorInfo's
- * data field.
- */
- virtual void registered(ExecutorDriver* driver,
- const ExecutorInfo& executorInfo,
- const FrameworkInfo& frameworkInfo,
- const SlaveInfo& slaveInfo) = 0;
-
- /**
- * Invoked when the executor re-registers with a restarted slave.
- */
- virtual void reregistered(ExecutorDriver* driver,
- const SlaveInfo& slaveInfo) = 0;
-
- /**
- * Invoked when the executor becomes "disconnected" from the slave
- * (e.g., the slave is being restarted due to an upgrade).
- */
- virtual void disconnected(ExecutorDriver* driver) = 0;
-
- /**
- * Invoked when a task has been launched on this executor (initiated
- * via Scheduler::launchTasks). Note that this task can be realized
- * with a thread, a process, or some simple computation, however, no
- * other callbacks will be invoked on this executor until this
- * callback has returned.
- */
- virtual void launchTask(ExecutorDriver* driver,
- const TaskInfo& task) = 0;
-
- /**
- * Invoked when a task running within this executor has been killed
- * (via SchedulerDriver::killTask). Note that no status update will
- * be sent on behalf of the executor, the executor is responsible
- * for creating a new TaskStatus (i.e., with TASK_KILLED) and
- * invoking ExecutorDriver::sendStatusUpdate.
- */
- virtual void killTask(ExecutorDriver* driver, const TaskID& taskId) = 0;
-
- /**
- * Invoked when a framework message has arrived for this
- * executor. These messages are best effort; do not expect a
- * framework message to be retransmitted in any reliable fashion.
- */
- virtual void frameworkMessage(ExecutorDriver* driver,
- const std::string& data) = 0;
-
- /**
- * Invoked when the executor should terminate all of it's currently
- * running tasks. Note that after a Mesos has determined that an
- * executor has terminated any tasks that the executor did not send
- * terminal status updates for (e.g., TASK_KILLED, TASK_FINISHED,
- * TASK_FAILED, etc) a TASK_LOST status update will be created.
- */
- virtual void shutdown(ExecutorDriver* driver) = 0;
-
- /**
- * Invoked when a fatal error has occured with the executor and/or
- * executor driver. The driver will be aborted BEFORE invoking this
- * callback.
- */
- virtual void error(ExecutorDriver* driver, const std::string& message) = 0;
-```
-
-
-h2. Install your Framework
-
-You need to put your framework somewhere that all slaves on the cluster can get it from. If you are running HDFS, you can put your executor into HDFS. Then, you tell Mesos where it is via the @ExecutorInfo@ parameter of @MesosSchedulerDriver@'s constructor (e.g. see src/examples/java/TestFramework.java for an example of this). ExecutorInfo is a a Protocol Buffer Message class (defined in @include/mesos/mesos.proto@), and you set its uri field to something like "HDFS://path/to/executor/". Also, you can pass the @frameworks_home@ configuration option (defaults to: @MESOS_HOME/frameworks@) to your @mesos-slave@ daemons when you launch them to specify where all of your framework executors are stored (e.g. on an NFS mount that is available to all slaves), then set @ExecutorInfo@ to be a relative path, and the slave will prepend the value of frameworks_home to the relative path provided.
-
-Once you are sure that your executors are available to the mesos-slaves, you should be able to run your scheduler, which will register with the Mesos master, and start receiving resource offers!
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/mesos/blob/5cd623e3/docs/Configuration.md
----------------------------------------------------------------------
diff --git a/docs/Configuration.md b/docs/Configuration.md
index 77e507c..423c53a 100644
--- a/docs/Configuration.md
+++ b/docs/Configuration.md
@@ -1,81 +1,88 @@
-# Configuration #
-If you have special compilation requirements, please refer to `./configure --help` when configuring Mesos.
+---
+layout: documentation
+---
-The Mesos master and slave can take a variety of configuration options through command-line arguments, or environment variables. A list of the available options can be seen by running `mesos-master --help` or `mesos-slave --help`. Each option can be set in three ways:
+# Mesos Configuration
- - By passing it to the binary using `--option_name=value`.
- - By setting the environment variable `MESOS_OPTION_NAME` (the option name with a `MESOS_` prefix added to it).
+The Mesos master and slave can take a variety of configuration options through command-line arguments, or environment variables. A list of the available options can be seen by running `mesos-master --help` or `mesos-slave --help`. Each option can be set in two ways:
-Configuration values are searched for first in the environment, then on the command-line.
+* By passing it to the binary using `--option_name=value`.
+* By setting the environment variable `MESOS_OPTION_NAME` (the option name with a `MESOS_` prefix added to it).
-## Important Options ##
+Configuration values are searched for first in the environment, then on the command-line.
-These are only a subset of the options. The following list may not match the version of mesos you are running, as the flags evolve from time to time in the code.
+**Important Options**
-**For now, the definitive source for which flags your version of Mesos supports can be found by running the binary with the flag `--help`, for example `bin/mesos-master --help`**. This will change as we add canonical documentation for each release.
+If you have special compilation requirements, please refer to `./configure --help` when configuring Mesos. Additionally, the documentation lists only a subset of the options. A definitive source for which flags your version of Mesos supports can be found by running the binary with the flag `--help`, for example `bin/mesos-master --help`.
-### Master and Slave Options ###
+## Master and Slave Options
- - `log_dir` : Directory for log files. If unspecified, nothing is written to disk.
+*These options can be supplied to both masters and slaves.*
- - `quiet` : Disable logging to stderr. [Default: false]
+* `log_dir` : Directory for log files. If unspecified, nothing is written to disk.
- - `ip` : IP address to listen on. [Optional]
+* `quiet` : Disable logging to stderr. [Default: false]
- - `port` : Port to listen on. [Default for master: 5050, Default for slave:5051]
+* `ip` : IP address to listen on. [Optional]
-### Master Options ###
+* `port` : Port to listen on. [Default for master: 5050, Default for slave:5051]
- - `zk` : ZooKeeper URL (used for leader election amongst multiple masters). May be one of:
- - zk://host1:port1,host2:port2,.../path
- - zk://username:password@host1:port1,host2:port2,.../path
- - file://path/to/file (where file contains one of the above)
+## Master Options
- - `roles` : A comma seperated list of the allocation roles
- that frameworks in this cluster may belong to.
+* `zk` : ZooKeeper URL (used for leader election amongst multiple masters). May be one of:
+ * zk://host1:port1,host2:port2,.../path
+ * zk://username:password@host1:port1,host2:port2,.../path
+ * file://path/to/file (where file contains one of the above)
- - `weights` : A comma seperated list of role/weight pairs
- of the form 'role=weight,role=weight'. Weights
- are used to indicate forms of priority.
+* `roles` : A comma seperated list of the allocation roles
+ that frameworks in this cluster may belong to.
-**For the complete list of master options: ./mesos-master.sh --help**
+* `weights` : A comma seperated list of role/weight pairs
+ of the form 'role=weight,role=weight'. Weights
+ are used to indicate forms of priority.
-### Slave Options ###
+## Slave Options
- - `resources` : Total consumable resources per slave, in
+* `resources` : Total consumable resources per slave, in
the form 'name(role):value;name(role):value...'.
- - NOTE: '(role)' is optional.
- - Ex: "cpus(role2):2;mem(role2):1024;cpus:1;mem:1024;disk:0"
-
- - `attributes` : Attributes for the machine. [Optional]
- - These are free-form in the same style as resources, ex: "rack:abc;kernel:2.6.44".
- - This information is provided to frameworks.
-
- - `work_dir` : Directory for the executor work directories. [Default: /tmp/mesos]
-
- - `isolation` : Isolation mechanism, one of: "process", "cgroups". [Default: process]
-
- - `cgroups_enable_cfs` : If using cgroups isolation, this enables hard limits on CPU resources.
-
- - `master` : May be one of:
- - zk://host1:port1,host2:port2,.../path
- - zk://username:password@host1:port1,host2:port2,.../path\n
- - file://path/to/file (where file contains one of the above)
-
- - `default_role` : Any resources in the --resources flag that
- omit a role, as well as any resources that
- are not present in --resources but that are
- automatically detected, will be assigned to
- this role. [Default: *]
-
- - `checkpoint` : Whether to checkpoint slave and frameworks information
- to disk.
- - This enables a restarted slave to recover status updates and reconnect
- with (--recover=reconnect) or kill (--recover=kill) old executors [Default: false]
-
- - `strict` : Whether to do recovery in strict mode [Default: true].
- - If strict=true, any and all recovery errors are considered fatal.
- - If strict=false, any errors (e.g., corruption in checkpointed data) during recovery are
- ignored and as much state as possible is recovered.
-
-**For the complete list of slave options: ./mesos-slave.sh --help**
+ * NOTE: '(role)' is optional.
+ * Ex: "cpus(role2):2;mem(role2):1024;cpus:1;mem:1024;disk:0"
+
+* `attributes` : Attributes for the machine. [Optional]
+ * These are free-form in the same style as resources, ex: "rack:abc;kernel:2.6.44".
+ * This information is provided to frameworks.
+
+* `work_dir` : Directory for the executor work directories. [Default: /tmp/mesos]
+ * `isolation` : Isolation mechanism, one of: "process", "cgroups". [Default: process]
+ * `cgroups_enable_cfs` : If using cgroups isolation, this enables hard limits on CPU resources.
+
+* `master` : May be one of:
+ * zk://host1:port1,host2:port2,.../path
+ * zk://username:password@host1:port1,host2:port2,.../path\n
+ * file://path/to/file (where file contains one of the above)
+
+* `default_role` : Any resources in the --resources flag that
+ omit a role, as well as any resources that
+ are not present in --resources but that are
+ automatically detected, will be assigned to
+ this role. [Default: *]
+
+* `checkpoint` : Whether to checkpoint slave and frameworks information
+ to disk.
+ * This enables a restarted slave to recover status updates and reconnect
+ with (--recover=reconnect) or kill (--recover=kill) old executors [Default: false]
+
+* `strict` : Whether to do recovery in strict mode [Default: true].
+ * If strict=true, any and all recovery errors are considered fatal.
+ * If strict=false, any errors (e.g., corruption in checkpointed data) during recovery are
+ ignored and as much state as possible is recovered.
+
+## Mesos Configuration Options
+
+The configure script itself accepts the following arguments to enable various options:
+
+* `--with-python-headers=DIR`: Find Python header files in `DIR` (to turn on Python support). Recommended.
+* `--with-webui`: Enable the Mesos web UI (which requires Python 2.6). Recommended.
+* `--with-java-home=DIR`: Enable Java application/framework support with a given installation of Java. Required for Hadoop and Spark.
+* `--with-java-headers=DIR`: Find Java header files (necessary for newer versions of OS X Snow Leopard).
+* `--with-included-zookeeper` or `--with-zookeeper=DIR`: Enable master fault-tolerance using an existing ZooKeeper installation or the version of ZooKeeper bundled with Mesos. For details, see [High Availability](High-Availability.md).
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/mesos/blob/5cd623e3/docs/Deploy-Scripts.md
----------------------------------------------------------------------
diff --git a/docs/Deploy-Scripts.md b/docs/Deploy-Scripts.md
new file mode 100644
index 0000000..cb04d5f
--- /dev/null
+++ b/docs/Deploy-Scripts.md
@@ -0,0 +1,16 @@
+---
+layout: documentation
+---
+
+# Deploy Scripts
+
+Mesos includes a set of scripts in @MESOS_HOME/deploy@ that can be used to deploy it on a cluster. To use these scripts, you need to create two configuration files: @MESOS_HOME/conf/masters@, which should list the hostname(s) of the node(s) you want to be your masters (one per line), and @MESOS_HOME/conf/slaves@, which should contain a list of hostnames for your slaves. You can then start a cluster with @deploy/start-mesos@ and stop it with @deploy/stop-mesos@.
+
+It is also possible to set environment variables, ulimits, etc that will affect the master and slave. by editing @MESOS_HOME/deploy/mesos-env.sh@. One particularly useful setting is @LIBPROCESS_IP@, which tells the master and slave binaries which IP address to bind to; in some installations, the default interface that the hostname resolves to is not the machine's external IP address, so you can set the right IP through this variable.
+
+Finally, the deploy scripts do not use ZooKeeper by default. If you want to use ZooKeeper (for multiple masters), you can do so by either editing @deploy/mesos-env.sh@ to set @MESOS_URL@ to a @zoo://@ or @zoofile://@ URL, or by editing @conf/mesos.conf@ to set the @url@ configuration parameter. Please see the "High Availability" documentation for details.
+
+## Notes
+
+* The deploy scripts assume that Mesos is located in the same directory on all nodes.
+* If you want to enable multiple Unix users to submit to the same cluster, you need to run the Mesos slaves as root (or possibly set the right attributes on the @mesos-slave@ binary). Otherwise, they will fail to @setuid@.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/mesos/blob/5cd623e3/docs/Deploy-Scripts.textile
----------------------------------------------------------------------
diff --git a/docs/Deploy-Scripts.textile b/docs/Deploy-Scripts.textile
deleted file mode 100644
index 0bfb692..0000000
--- a/docs/Deploy-Scripts.textile
+++ /dev/null
@@ -1,9 +0,0 @@
-Mesos includes a set of scripts in @MESOS_HOME/deploy@ that can be used to deploy it on a cluster. To use these scripts, you need to create two configuration files: @MESOS_HOME/conf/masters@, which should list the hostname(s) of the node(s) you want to be your masters (one per line), and @MESOS_HOME/conf/slaves@, which should contain a list of hostnames for your slaves. You can then start a cluster with @deploy/start-mesos@ and stop it with @deploy/stop-mesos@.
-
-It is also possible to set environment variables, ulimits, etc that will affect the master and slave. by editing @MESOS_HOME/deploy/mesos-env.sh@. One particularly useful setting is @LIBPROCESS_IP@, which tells the master and slave binaries which IP address to bind to; in some installations, the default interface that the hostname resolves to is not the machine's external IP address, so you can set the right IP through this variable.
-
-Finally, the deploy scripts do not use ZooKeeper by default. If you want to use ZooKeeper (for multiple masters), you can do so by either editing @deploy/mesos-env.sh@ to set @MESOS_URL@ to a @zoo://@ or @zoofile://@ URL, or by editing @conf/mesos.conf@ to set the @url@ configuration parameter. Please see "Using ZooKeeper":Using-ZooKeeper.textile for details.
-
-h1. Notes
-* The deploy scripts assume that Mesos is located in the same directory on all nodes.
-* If you want to enable multiple Unix users to submit to the same cluster, you need to run the Mesos slaves as root (or possibly set the right attributes on the @mesos-slave@ binary). Otherwise, they will fail to @setuid@.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/mesos/blob/5cd623e3/docs/EC2-Scripts.md
----------------------------------------------------------------------
diff --git a/docs/EC2-Scripts.md b/docs/EC2-Scripts.md
new file mode 100644
index 0000000..a09c73c
--- /dev/null
+++ b/docs/EC2-Scripts.md
@@ -0,0 +1,59 @@
+---
+layout: documentation
+---
+
+# EC2 Scripts
+
+The @mesos-ec2@ script located in the Mesos's @ec2@ directory allows you to launch, manage and shut down Mesos clusters on Amazon EC2. You don't need to build Mesos to use this script -- you just need Python 2.6+ installed.
+
+@mesos-ec2@ is designed to manage multiple named clusters. You can launch a new cluster (telling the script its size and giving it a name), shutdown an existing cluster, or log into a cluster. Each cluster is identified by placing its machines into EC2 security groups whose names are derived from the name of the cluster. For example, a cluster named @test@ will contain a master node in a security group called @test-master@, and a number of slave nodes in a security group called @test-slaves@. The @mesos-ec2@ script will create these security groups for you based on the cluster name you request. You can also use them to identify machines belonging to each cluster in the EC2 Console or ElasticFox.
+
+This guide describes how to get set up to run clusters, how to launch clusters, how to run jobs on them, and how to shut them down.
+
+## Before You Start
+
+* Create an Amazon EC2 key pair for yourself. This can be done by logging into your Amazon Web Services account through the "AWS console":http://aws.amazon.com/console/, clicking Key Pairs on the left sidebar, and creating and downloading a key. Make sure that you set the permissions for the private key file to @600@ (i.e. only you can read and write it) so that @ssh@ will work.
+* Whenever you want to use the @mesos-ec2@ script, set the environment variables @AWS_ACCESS_KEY_ID@ and @AWS_SECRET_ACCESS_KEY@ to your Amazon EC2 access key ID and secret access key. These can be obtained from the "AWS homepage":http://aws.amazon.com/ by clicking Account > Security Credentials > Access Credentials.
+
+## Launching a Cluster
+
+* Go into the @ec2@ directory in the release of Mesos you downloaded.
+* Run @./mesos-ec2 -k <keypair> -i <key-file> -s <num-slaves> launch <cluster-name>@, where @<keypair>@ is the name of your EC2 key pair (that you gave it when you created it), @<key-file>@ is the private key file for your key pair, @<num-slaves>@ is the number of slave nodes to launch (try 1 at first), and @<cluster-name>@ is the name to give to your cluster.
+* After everything launches, check that Mesos is up and sees all the slaves by going to the Mesos Web UI link printed at the end of the script (@http://<master-hostname>:8080@).
+
+You can also run @./mesos-ec2 --help@ to see more usage options. The following options are worth pointing out:
+
+* @--instance-type=<INSTANCE_TYPE>@ can be used to specify an EC2 instance type to use. For now, the script only supports 64-bit instance types, and the default type is @m1.large@ (which has 2 cores and 7.5 GB RAM). Refer to the Amazon pages about "EC2 instance types":http://aws.amazon.com/ec2/instance-types and "EC2 pricing":http://aws.amazon.com/ec2/#pricing for information about other instance types.
+* @--zone=<EC2_ZONE>@ can be used to specify an EC2 availability zone to launch instances in. Sometimes, you will get an error because there is not enough capacity in one zone, and you should try to launch in another. This happens mostly with the @m1.large@ instance types; extra-large (both @m1.xlarge@ and @c1.xlarge@) instances tend to be more available.
+* @--download=git@ will tell the instances to download the latest release of Mesos from the github git repository.
+* @--ft=<NUM_MASTERS>@ can be used to run Mesos in fault tolerant (FT) mode by specifying @NUM_MASTERS@ > 1.
+* If one of your launches fails due to e.g. not having the right permissions on your private key file, you can run @launch@ with the @--resume@ option to restart the setup process on an existing cluster.
+
+## Running Jobs
+
+* Go into the @ec2@ directory in the release of Mesos you downloaded.
+* Run @./mesos-ec2 -k <keypair> -i <key-file> login <cluster-name>@ to SSH into the cluster, where @<keypair>@ and @<key-file>@ are as above. (This is just for convenience; you could also use Elasticfox or the EC2 console.)
+* Copy your code to all the nodes. To do this, you can use the provided script @~/mesos-ec2/copy-dir@, which, given a directory path, RSYNCs it to the same location on all the slaves.
+* If your job needs to access large datasets, the fastest way to do that is to load them from Amazon S3 or an Amazon EBS device into an instance of the Hadoop Distributed File System (HDFS) on your nodes. The @mesos-ec2@ script already sets up a HDFS instance for you. It's installed in @/root/ephemeral-hdfs@, and can be accessed using the @bin/hadoop@ script in that directory. Note that the data in this HDFS goes away when you stop and restart a machine.
+* There is also a _persistent HDFS_ instance in @/root/presistent-hdfs@ that will keep data across cluster restarts. Typically each node has relatively little space of persistent data (about 3 GB), but you can use the @--ebs-vol-size@ option to @mesos-ec2@ to attach a persistent EBS volume to each node for storing the persistent HDFS.
+
+If you get an "Executor on slave X disconnected" error when running your framework, you probably haven't copied your code the slaves. Use the @~/mesos-ec2/copy-dir@ script to do that. If you keep getting the error, though, look at the slave's logs for that framework using the Mesos web UI. Please see "logging and debugging":Logging-and-Debugging.textile for details.
+
+## Terminating a Cluster
+
+_*Note that there is no way to recover data on EC2 nodes after shutting them down! Make sure you have copied everything important off the nodes before stopping them.*_
+
+* Go into the @ec2@ directory in the release of Mesos you downloaded.
+* Run @./mesos-ec2 destroy <cluster-name>@.
+
+## Pausing and Restarting EBS-Backed Clusters
+
+The @mesos-ec2@ script also supports pausing a cluster if you are using EBS-backed virtual machines (which all of our machine images are by default). In this case, the VMs are stopped but not terminated, so they _*lose all data on ephemeral disks (/mnt, ephemeral-hdfs)*_ but keep the data in their root partitions and their @persistent-hdfs@. Stopped machines will not cost you any EC2 cycles, but _*will*_ continue to cost money for EBS storage.
+
+* To stop one of your clusters, go into the @ec2@ directory and run @./mesos-ec2 stop <cluster-name>@.
+* To restart it later, run @./mesos-ec2 -i <key-file> start <cluster-name>@.
+* To ultimately destroy the cluster and stop consuming EBS space, run @./mesos-ec2 destroy <cluster-name>@ as described in the previous section.
+
+## Limitations
+
+* The @mesos-ec2@ script currently does not use the "deploy scripts":Deploy-Scripts.textile included with Mesos to manage its clusters. This will likely be fixed in the future.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/mesos/blob/5cd623e3/docs/EC2-Scripts.textile
----------------------------------------------------------------------
diff --git a/docs/EC2-Scripts.textile b/docs/EC2-Scripts.textile
deleted file mode 100644
index 7a1d1da..0000000
--- a/docs/EC2-Scripts.textile
+++ /dev/null
@@ -1,52 +0,0 @@
-The @mesos-ec2@ script located in the Mesos's @ec2@ directory allows you to launch, manage and shut down Mesos clusters on Amazon EC2. You don't need to build Mesos to use this script -- you just need Python 2.6+ installed.
-
-@mesos-ec2@ is designed to manage multiple named clusters. You can launch a new cluster (telling the script its size and giving it a name), shutdown an existing cluster, or log into a cluster. Each cluster is identified by placing its machines into EC2 security groups whose names are derived from the name of the cluster. For example, a cluster named @test@ will contain a master node in a security group called @test-master@, and a number of slave nodes in a security group called @test-slaves@. The @mesos-ec2@ script will create these security groups for you based on the cluster name you request. You can also use them to identify machines belonging to each cluster in the EC2 Console or ElasticFox.
-
-This guide describes how to get set up to run clusters, how to launch clusters, how to run jobs on them, and how to shut them down.
-
-h1. Before You Start
-
-* Create an Amazon EC2 key pair for yourself. This can be done by logging into your Amazon Web Services account through the "AWS console":http://aws.amazon.com/console/, clicking Key Pairs on the left sidebar, and creating and downloading a key. Make sure that you set the permissions for the private key file to @600@ (i.e. only you can read and write it) so that @ssh@ will work.
-* Whenever you want to use the @mesos-ec2@ script, set the environment variables @AWS_ACCESS_KEY_ID@ and @AWS_SECRET_ACCESS_KEY@ to your Amazon EC2 access key ID and secret access key. These can be obtained from the "AWS homepage":http://aws.amazon.com/ by clicking Account > Security Credentials > Access Credentials.
-
-h1. Launching a Cluster
-
-* Go into the @ec2@ directory in the release of Mesos you downloaded.
-* Run @./mesos-ec2 -k <keypair> -i <key-file> -s <num-slaves> launch <cluster-name>@, where @<keypair>@ is the name of your EC2 key pair (that you gave it when you created it), @<key-file>@ is the private key file for your key pair, @<num-slaves>@ is the number of slave nodes to launch (try 1 at first), and @<cluster-name>@ is the name to give to your cluster.
-* After everything launches, check that Mesos is up and sees all the slaves by going to the Mesos Web UI link printed at the end of the script (@http://<master-hostname>:8080@).
-
-You can also run @./mesos-ec2 --help@ to see more usage options. The following options are worth pointing out:
-* @--instance-type=<INSTANCE_TYPE>@ can be used to specify an EC2 instance type to use. For now, the script only supports 64-bit instance types, and the default type is @m1.large@ (which has 2 cores and 7.5 GB RAM). Refer to the Amazon pages about "EC2 instance types":http://aws.amazon.com/ec2/instance-types and "EC2 pricing":http://aws.amazon.com/ec2/#pricing for information about other instance types.
-* @--zone=<EC2_ZONE>@ can be used to specify an EC2 availability zone to launch instances in. Sometimes, you will get an error because there is not enough capacity in one zone, and you should try to launch in another. This happens mostly with the @m1.large@ instance types; extra-large (both @m1.xlarge@ and @c1.xlarge@) instances tend to be more available.
-* @--download=git@ will tell the instances to download the latest release of Mesos from the github git repository.
-* @--ft=<NUM_MASTERS>@ can be used to run Mesos in fault tolerant (FT) mode by specifying @NUM_MASTERS@ > 1.
-* If one of your launches fails due to e.g. not having the right permissions on your private key file, you can run @launch@ with the @--resume@ option to restart the setup process on an existing cluster.
-
-h1. Running Jobs
-
-* Go into the @ec2@ directory in the release of Mesos you downloaded.
-* Run @./mesos-ec2 -k <keypair> -i <key-file> login <cluster-name>@ to SSH into the cluster, where @<keypair>@ and @<key-file>@ are as above. (This is just for convenience; you could also use Elasticfox or the EC2 console.)
-* Copy your code to all the nodes. To do this, you can use the provided script @~/mesos-ec2/copy-dir@, which, given a directory path, RSYNCs it to the same location on all the slaves.
-* If your job needs to access large datasets, the fastest way to do that is to load them from Amazon S3 or an Amazon EBS device into an instance of the Hadoop Distributed File System (HDFS) on your nodes. The @mesos-ec2@ script already sets up a HDFS instance for you. It's installed in @/root/ephemeral-hdfs@, and can be accessed using the @bin/hadoop@ script in that directory. Note that the data in this HDFS goes away when you stop and restart a machine.
-* There is also a _persistent HDFS_ instance in @/root/presistent-hdfs@ that will keep data across cluster restarts. Typically each node has relatively little space of persistent data (about 3 GB), but you can use the @--ebs-vol-size@ option to @mesos-ec2@ to attach a persistent EBS volume to each node for storing the persistent HDFS.
-
-If you get an "Executor on slave X disconnected" error when running your framework, you probably haven't copied your code the slaves. Use the @~/mesos-ec2/copy-dir@ script to do that. If you keep getting the error, though, look at the slave's logs for that framework using the Mesos web UI. Please see "logging and debugging":Logging-and-Debugging.textile for details.
-
-h1. Terminating a Cluster
-
-_*Note that there is no way to recover data on EC2 nodes after shutting them down! Make sure you have copied everything important off the nodes before stopping them.*_
-
-* Go into the @ec2@ directory in the release of Mesos you downloaded.
-* Run @./mesos-ec2 destroy <cluster-name>@.
-
-h1. Pausing and Restarting EBS-Backed Clusters
-
-The @mesos-ec2@ script also supports pausing a cluster if you are using EBS-backed virtual machines (which all of our machine images are by default). In this case, the VMs are stopped but not terminated, so they _*lose all data on ephemeral disks (/mnt, ephemeral-hdfs)*_ but keep the data in their root partitions and their @persistent-hdfs@. Stopped machines will not cost you any EC2 cycles, but _*will*_ continue to cost money for EBS storage.
-
-* To stop one of your clusters, go into the @ec2@ directory and run @./mesos-ec2 stop <cluster-name>@.
-* To restart it later, run @./mesos-ec2 -i <key-file> start <cluster-name>@.
-* To ultimately destroy the cluster and stop consuming EBS space, run @./mesos-ec2 destroy <cluster-name>@ as described in the previous section.
-
-h1. Limitations
-
-* The @mesos-ec2@ script currently does not use the "deploy scripts":Deploy-Scripts.textile included with Mesos to manage its clusters. This will likely be fixed in the future.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/mesos/blob/5cd623e3/docs/Event-history.md
----------------------------------------------------------------------
diff --git a/docs/Event-history.md b/docs/Event-history.md
deleted file mode 100644
index 5419800..0000000
--- a/docs/Event-history.md
+++ /dev/null
@@ -1,3 +0,0 @@
-This is a feature [currently being developed](https://github.com/mesos/mesos/tree/andyk-event-history-nowebui) that will allow capture the history of significant events in the context of all things Mesos (e.g. Jobs/tasks registering, starting, failing, etc.) into a variety of places where it can be used in the future.
-
-This is still under development.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/mesos/blob/5cd623e3/docs/Getting-Started.md
----------------------------------------------------------------------
diff --git a/docs/Getting-Started.md b/docs/Getting-Started.md
index 48ceb56..3986dab 100644
--- a/docs/Getting-Started.md
+++ b/docs/Getting-Started.md
@@ -1,7 +1,10 @@
-# Getting Started #
+---
+layout: gettingstarted
+---
+# Getting Started
-### Downloading Mesos ###
+## Downloading Mesos
There are different ways you can get Mesos:
@@ -14,8 +17,7 @@ There are different ways you can get Mesos:
$ git clone http://git-wip-us.apache.org/repos/asf/mesos.git
-
-### System Requirements ###
+## System Requirements
- Mesos runs on Linux and Mac OSX.
@@ -48,7 +50,6 @@ If you are building from git repository, you will need to additionally install t
# Install libtool.
$ sudo apt-get install libtool
-
***NOTES***
> 1. The build process attempts to guess where your Java include directory is, but if you have set the `$JAVA_HOME` environment variable, it will use `$JAVA_HOME/include`, which may not be correct (or exist) on your machine (in which case you will see an error such as: `configure: error: failed to build against JDK (using libtool)`). If this is the case, we suggest you unset the `JAVA_HOME` environment variable.
@@ -56,7 +57,7 @@ If you are building from git repository, you will need to additionally install t
> 2. Mesos is currently being developed/tested/supported on 64 Bit machines only.
-### Building Mesos ###
+## Building Mesos
# Change working directory.
$ cd mesos
@@ -76,10 +77,7 @@ If you are building from git repository, you will need to additionally install t
# Install (***Optional***).
$ make install
-
-
-### Examples ###
-
+## Examples
Mesos comes bundled with example frameworks written in `C++`, `Java` and `Python`.
# Change into build directory.
@@ -103,5 +101,4 @@ Mesos comes bundled with example frameworks written in `C++`, `Java` and `Python
# Run Python framework (***Exits after successfully running some tasks.***).
$ ./src/examples/python/test-framework 127.0.0.1:5050
-
-> NOTE: To build the example frameworks, make sure you build the test suite by doing `make check`.
+*NOTE: To build the example frameworks, make sure you build the test suite by doing `make check`.*
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/mesos/blob/5cd623e3/docs/Hadoop-demo-VM.md
----------------------------------------------------------------------
diff --git a/docs/Hadoop-demo-VM.md b/docs/Hadoop-demo-VM.md
deleted file mode 100644
index 6002b55..0000000
--- a/docs/Hadoop-demo-VM.md
+++ /dev/null
@@ -1,79 +0,0 @@
-## Mesos VMWare Image with Hadoop pre-installed
-
-***
-Last Updated: August 2011
-
-[DOWNLOAD](http://amplab.cs.berkeley.edu/downloads/mesos/mesos-demo.tar.bz2)
-
-**Note: This VM download is approximately 1.7GB. Feel Free to mirror internally or externally to minimize bandwidth usage. The Uncompressed version of VM required 6GB of disk space.**
-
-To make it easy for you to get started with Apache Mesos, we created a virtual machine with everything you need. our VM runs Ubuntu 10.04 LTS - Long-term support 64-bit (Lucid Lynx) and Mesos with Apache Hadoop 0.20.2.
-
-To launch the VMWare image, you will either need [VMware Player for Windows and Linux](http://www.vmware.com/go/downloadplayer/), or [VMware Fusion for Mac](http://www.vmware.com/products/fusion/). (Note that VMware Fusion only works on Intel architectures, so older Macs with PowerPC processors cannot run the training VM.)
-
-Once you launch the VM, log in with the following account details:
-
- - username: hadoop
- - password: hadoop
-
-The **hadoop** account has *sudo* privileges in the VM.
-
-### 1. Testing Mesos
-* run ` ~/mesos$ bin/tests/all-tests `
-
-```
-~/mesos$ bin/tests/all-tests
-[==========] Running 61 tests from 6 test cases.
-[----------] Global test environment set-up.
-[----------] 18 tests from MasterTest
-[ RUN ] MasterTest.ResourceOfferWithMultipleSlaves
-[ OK ] MasterTest.ResourceOfferWithMultipleSlaves (33 ms)
-[ RUN ] MasterTest.ResourcesReofferedAfterReject
-[ OK ] MasterTest.ResourcesReofferedAfterReject (3 ms)
-
-[ ... trimmed ... ]
-
-[ RUN ] MasterTest.MultipleExecutors
-[ OK ] MasterTest.MultipleExecutors (2 ms)
-[----------] 18 tests from MasterTest (38 ms total)
-
-[----------] Global test environment tear-down
-[==========] 61 tests from 6 test cases ran. (17633 ms total)
-[ PASSED ] 61 tests.
- YOU HAVE 3 DISABLED TESTS
-```
-
-### 2. Start all your frameworks!
-* Start Mesos's Master:
-` ~/mesos$ bin/mesos-master &`
-* Start Mesos's Slave:
-` ~/mesos$ bin/mesos-slave --url=mesos://master@localhost:5050 &`
-* Start Hadoop's namenode:
-` ~/mesos/frameworks/hadoop-0.20.2$ bin/hadoop namenode &`
-* Start Hadoop's datanode:
-` ~/mesos/frameworks/hadoop-0.20.2$ bin/hadoop datanode &`
-* Start Hadoop's jobtracker:
-` ~/mesos/frameworks/hadoop-0.20.2$ bin/hadoop jobtracker &`
-
-### 3. Run the MapReduce job:
- We will now run your first Hadoop MapReduce job. We will use the [WordCount](http://wiki.apache.org/hadoop/WordCount) example job which reads text files and counts how often words occur. The input is text files and the output is text files, each line of which contains a word and the count of how often it occurred, separated by a tab.
-
-* Run the "wordcount" example MapReduce job:
- ` ~/mesos/frameworks/hadoop-0.20.2$ bin/hadoop jar build/hadoop-0.20.3-dev-examples.jar wordcount /user/hadoop/g /user/hadoop/output`
-* You will see something like the following:
-
-
-11/07/19 15:34:29 INFO input.FileInputFormat: Total input paths to process : 6
-11/07/19 15:34:29 INFO mapred.JobClient: Running job: job_201107191533_0001
-11/07/19 15:34:30 INFO mapred.JobClient: map 0% reduce 0%
-11/07/19 15:34:43 INFO mapred.JobClient: map 16% reduce 0%
-
-[ ... trimmed ... ]
-
-Click the Firefox Web Browser on the Panel to view Mesos's documentation.
-The browser also provides the following bookmarks:
-
- * [http://localhost:50030](http://localhost:50030) - web UI for MapReduce job tracker(s)
- * [http://localhost:50060](http://localhost:50060) - web UI for task tracker(s)
- * [http://localhost:50070](http://localhost:50070) - web UI for HDFS name node(s)
- * [http://localhost:8080](http://localhost:8080) - web UI for Mesos master
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/mesos/blob/5cd623e3/docs/High-Availability.md
----------------------------------------------------------------------
diff --git a/docs/High-Availability.md b/docs/High-Availability.md
index 4235817..77af5d5 100644
--- a/docs/High-Availability.md
+++ b/docs/High-Availability.md
@@ -1,41 +1,42 @@
-High Availability
-=========
+---
+layout: documentation
+---
+
+# High Availability
+
Mesos can run in high-availability mode, in which multiple Mesos masters run simultaneously. In this mode there is only one active master, and the others masters act as stand-by replacements. These non-active masters will take over if the active master fails.
-Mesos uses Apache ZooKeeper in to co-ordinate the leader election of masters.
+Mesos uses Apache ZooKeeper in to coordinate the leader election of masters.
Running mesos in this mode requires ZooKeeper. Mesos is automatically built with an included ZooKeeper client library.
Leader election and detection in Mesos is done via ZooKeeper. See: http://zookeeper.apache.org/doc/trunk/recipes.html#sc_leaderElection for more information how ZooKeeper leader election works.
-Use
----------
+## Use
First, a ZooKeeper cluster must be running, and a znode should be created for exclusive use by Mesos:
- - Ensure the ZooKeeper cluster is up and running.
- - Create a znode for exclusive use by mesos. The znode path will need to be provided to all slaves / scheduler drivers.
+* Ensure the ZooKeeper cluster is up and running.
+* Create a znode for exclusive use by mesos. The znode path will need to be provided to all slaves / scheduler drivers.
In order to spin up a Mesos cluster using multiple masters for fault-tolerance:
- - Start the mesos-master binaries using the `--zk` flag, e.g. `--zk=zk://host1:port1/path,host2:port2/path,...`
- - Start the mesos-slave binaries with `--master=zk://host1:port1/path,host2:port2/path,...`
- - Start any framework schedulers using the same zk path, the SchedulerDriver must be constructed with this path.
+* Start the mesos-master binaries using the `--zk` flag, e.g. `--zk=zk://host1:port1/path,host2:port2/path,...`
+* Start the mesos-slave binaries with `--master=zk://host1:port1/path,host2:port2/path,...`
+* Start any framework schedulers using the same zk path, the SchedulerDriver must be constructed with this path.
Refer to the Scheduler API for how to deal with leadership changes.
-Semantics
----------
+## Semantics
The detector is implemented in the `src/detector` folder. In particular, we watch for several ZooKeeper session events:
- - Connection
- - Reconnection
- - Session Expiration
- - ZNode creation, deletion, updates
+* Connection
+* Reconnection
+* Session Expiration
+* ZNode creation, deletion, updates
We also explicitly timeout our sessions, when disconnected from ZooKeeper for an amount of time, see: `ZOOKEEPER_SESSION_TIMEOUT`. This is because the ZooKeeper client libraries only notify of session expiration upon reconnection. These timeouts are of particular interest in the case of network partitions.
-Network Partitions
-------------------
+## Network Partitions
When a network partition occurs, if a particular component is disconnected from ZooKeeper, the Master Detector of the partitioned component will induce a timeout event. This causes the component to be notified that there is no leading master.
When slaves are master-less, they ignore incoming messages from masters to ensure that we don't act on a non-leading master's decision.
@@ -44,6 +45,6 @@ When masters enter a leader-less state, they commit suicide.
The following semantics are enforced:
- - If a slave is partitioned from the master, it will fail health-checks. The master will mark the slave as deactivated and send its tasks to LOST.
- - Deactivated slaves may not re-register with the master, and are instructed to shut down upon any further communication after deactivation.
- - When a slave is partitioned from ZooKeeper, it will enter a master-less state. It will ignore any master messages until a new master is elected.
+* If a slave is partitioned from the master, it will fail health-checks. The master will mark the slave as deactivated and send its tasks to LOST.
+* Deactivated slaves may not re-register with the master, and are instructed to shut down upon any further communication after deactivation.
+* When a slave is partitioned from ZooKeeper, it will enter a master-less state. It will ignore any master messages until a new master is elected.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/mesos/blob/5cd623e3/docs/Home.md
----------------------------------------------------------------------
diff --git a/docs/Home.md b/docs/Home.md
index df6286b..0cbe85c 100644
--- a/docs/Home.md
+++ b/docs/Home.md
@@ -1,109 +1,44 @@
-# Overview of Mesos
+---
+layout: documentation
+---
-Mesos is a cluster management platform that provides resource sharing and isolation across distributed applications. For example, [Hadoop MapReduce](http://hadoop.apache.org), [MPI](http://www.mcs.anl.gov/research/projects/mpich2/), [Hypertable](http://hypertable.org), and [Spark](http://github.com/mesos/spark/wiki) (a new MapReduce-like framework from the Mesos team that supports low-latency interactive and iterative jobs) can run on Mesos.
+# Documentation
-#### Mesos Use-cases
-* Run Hadoop, MPI, Spark and other cluster applications on a dynamically shared pool of machines
-* Run **multiple instances of Hadoop** on the same cluster, e.g. separate instances for production and experimental jobs, or even multiple versions of Hadoop
-* Build new cluster applications without having to reinvent low-level facilities for running tasks on different nodes, monitoring them, etc., and have your applications coexist with existing ones
+## Users Documentation
-#### Features of Mesos
-* Master fault tolerance using ZooKeeper
-* Isolation between tasks using Linux Containers
-* Memory and CPU aware allocation
-* Efficient fine-grained resource sharing abstraction (<i>resource offers</i>) that allows applications to achieve app-specific scheduling goals (e.g. Hadoop optimizes for data locality)
-<br/>
-Visit [mesosproject.org](http://mesosproject.org) for more details about Mesos.
+* [Deploy Scripts](Deploy-Scripts.md) for launching a Mesos cluster on a set of machines.
+* [EC2 scripts](EC2-Scripts.md) for launching a Mesos cluster on Amazon EC2.
+* [Logging and Debugging](Logging-and-Debugging.md) -- viewing Mesos and framework logs.
+* [High Availability](High-Availability) for running multiple masters simultaneously.
+* [Slave Recovery](Slave-Recovery.md)
+* [Using Linux Containers](Using-Linux-Containers.md) for resource isolation on slaves.
+* [Using Mesos Submit Tool](Using-the-mesos-submit-tool.md) for launching a framework with a single task.
-_**Please note that Mesos is still in beta. Though the current version is in use in production at Twitter, it may have some stability issues in certain environments.**_
+## Running Mesos Frameworks
-#### What you'll find on this page
-1. Quick-start Guides
-1. System Requirements
-1. Downloading Mesos
-1. Building Mesos
-1. Testing the Build
-1. Deploying to a Cluster
-1. Where to Go From Here
+* [Running Hadoop on Mesos](https://github.com/mesos/hadoop)
+* [Running Spark on Mesos](https://cwiki.apache.org/confluence/display/SPARK/)
+* [Running Torque or MPI on Mesos](Running-torque-or-mpi-on-mesos.md)
+* [Running Jenkins on Mesos](https://github.com/jenkinsci/mesos-plugin)
+* [Running Chronos on Mesos](https://github.com/airbnb/chronos)
+* [Running Marathon on Mesos](https://github.com/mesosphere/marathon)
+* [Using Mesos with Hypertable on EC2](https://code.google.com/p/hypertable/wiki/Mesos)
-# System Requirements
+## Developing Mesos Frameworks and Applications
-Mesos runs on Linux and Mac OS X, and has previously also been tested on OpenSolaris. You will need the following packages to run it:
+* [Mesos Architecture](Mesos-Architecture.md) providing an overview of Mesos concepts.
+* [Mesos Code Internals](Mesos-Code-Internals.md) overview of the codebase and internal organization.
+* [Framework Development Guide](App-Framework-Development-Guide.md) describes how to build applications on top of Mesos.
+* [Mesos Command-Line Flags](Configuration.md) detailing various flags that can be set.
-* g++ 4.1 or higher.
-* Python 2.6 (for the Mesos web UI). On Mac OS X 10.6 or earlier, get Python from [MacPorts](http://www.macports.org/) via `port install python26`, because OS X's version of Python is not 64-bit.
-* Python 2.6 developer packages (on red-hat - sudo yum install python26-devel python-devel)
-* cppunit (for building zookeeper) (on red-hat - sudo yum install cppunit-devel)
-* Java JDK 1.6 or higher. Mac OS X 10.6 users will need to install the JDK from http://connect.apple.com/ and set `JAVA_HEADERS=/System/Library/Frameworks/JavaVM.framework/Versions/A/Headers`.
+## Contributing to Mesos
-# Downloading and Building Mesos
+* [Mesos Developers Guide](Mesos-developers-guide.md) includes resources for developers contributing to Mesos and the process of submitting patches for review.
+* [C++ Style Guide](Mesos-c++-style-guide.md)
+* [Mesos Development Road Map](Mesos-Roadmap.md)
-Mesos uses the standard GNU build tools. See the section farther below for instructions for checking out and building the source code via source control.
+## More Info about Mesos
-To get running with Mesos version 0.9.0-incubating, our most recently release:
-
-1. [Download Mesos 0.9.0-incubating](http://www.apache.org/dyn/closer.cgi/incubator/mesos/mesos-0.9.0-incubating/)
-1. Run configure (there are a few different helper scripts that wrap the `configure` script called configure.<type-of-os>)
- 1. In OS X: run `./configure.macosx`. If you're running Mountain Lion, you may need to follow the instructions [here](https://issues.apache.org/jira/browse/MESOS-261?focusedCommentId=13447058&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13447058) and [here](https://issues.apache.org/jira/browse/MESOS-285).
- 1. In Linux, you can probably just run `./configure`, which includes the web UI and included Zookeeper. Advanced users may want to exclude these options or include other options, see [Mesos Command-Line Flags](Configuration.textile).
-1. run `make`
-
-### NOTES:
-* In the SVN trunk branch since Jan 19 2012 (when Mesos switched fully to the GNU Autotools build system), the build process attempts to guess where your Java include directory is, but if you have set the $JAVA_HOME environment variable, it will use $JAVA_HOME/include, which may not be correct (or exist) on your machine (in which case you will see an error such as: "configure: error: failed to build against JDK (using libtool)"). If this is the case, we suggest you unset the JAVA_HOME environment variable.
-* `configure` may print a warning at the end about "unrecognized options: --with-java-home, ...". This comes from one of the nested `configure` scripts that we call, so it doesn't mean that your options were ignored.
-* (NOT SURE IF THIS IS STILL RELEVANT) On 32-bit platforms, you should set `CXXFLAGS="-march=i486"` when running `configure` to ensure certain atomic instructions can be used.
-
-# Checking Mesos Source Code out of Git or SVN
-
-Currently, you can obtain the current Mesos development HEAD by checking it out from either the Apache SVN or Apache Git repository (the git repo is a mirror of the SVN repo)
-* For SVN, use: `svn co https://svn.apache.org/repos/asf/incubator/mesos/trunk mesos-trunk`
-* For git, use: `git clone git://git.apache.org/mesos.git`
-
-# Running Example Frameworks and Testing the Build
-
-Currently, in order to run the example frameworks (in src/examples), you must first build the test suite, as instructed below.
-
-After you build Mesos by running the `make` command, you can build and run its example frameworks and unit tests (which use the example frameworks) by issuing the `make check` command from the directory where you ran the `make` command.
-
-After you have done this, you can also set up a small Mesos cluster and run a job on it as follows:
-
-1. Go into the directory where you built Mesos.
-1. Type `bin/mesos-master.sh` to launch the master.
-1. Take note of the IP and port that the master is running on, which will look something like <code>192.168.0.1:5050</code>. <i>Note: In this example the IP address of master is 192.168.0.1, and the port is 5050. We will continue to use the URL shown here for the rest of this example; however when you run the following commands replace all instances of it with the URL of your master.</i>
-1. URL of master: <code>192.168.0.1:5050</code>
-1. View the master's web UI at `http://[hostname of master]:5050`.
-1. Launch a slave by typing <code>bin/mesos-slave.sh --master=192.168.0.1:5050</code>. The slave will show up on the master's web UI if you refresh it.
-1. Run the C++ test framework (a sample that just runs five tasks on the cluster) using <code>src/test-framework 192.168.0.1:5050</code>. It should successfully exit after running five tasks.
-1. You can also try the example python or Java frameworks, with commands like the following:
- 2. `src/examples/java/test-framework 192.168.0.1:5050`
- 2. `src/examples/python/test-framework 192.168.0.1:5050`
-
-# Deploying to a Cluster
-
-To run Mesos on more than one machine, you have multiple options:
-
-* To launch a cluster on a private cluster that you own, use Mesos's [deploy scripts](Deploy-Scripts.textile)
-* To launch a cluster on Amazon EC2, you can use the Mesos [EC2 scripts](EC2-Scripts.textile)
-
-# Where to Go From Here
-
-* [Mesos Architecture](Mesos-Architecture.md) -- an overview of Mesos concepts.
-* [Mesos Developers Guide](Mesos-developers-guide.md) -- resources for developers contributing to Mesos. Style guides, tips for working with SVN/git, and more!
-* [App/Framework Development Guide](App-Framework-development-guide.textile) -- learn how to build applications on top of Mesos.
-* [Mesos Command-Line Flags](Configuration.textile) -- a guide to the various flags that can be set.
-* Mesos system feature guides:
- * [Deploy Scripts](Deploy-Scripts.textile) for launching a Mesos cluster on a set of machines.
- * [EC2 scripts](EC2-Scripts.textile) for launching a Mesos cluster on Amazon EC2.
- * [Logging and Debugging](Logging-and-Debugging.textile) -- viewing Mesos and framework logs.
- * [Using ZooKeeper](Using-ZooKeeper.textile) for master fault-tolerance.
- * [Using Linux Containers](Using-Linux-Containers.textile) for resource isolation on slaves.
-* Guide to running existing frameworks:
- * [Running Spark on Mesos](https://github.com/mesos/spark/wiki)
- * [Using the Mesos Submit tool](Using-the-mesos-submit-tool.md)
- * [Using Mesos with Hypertable on EC2](http://code.google.com/p/hypertable/wiki/Mesos) - Hypertable is a distributed database platform.
- * [Running Hadoop on Mesos](Running-Hadoop-on-Mesos.md)
- * [Running a web application farm on Mesos](Running-a-web-application-farm-on-mesos.textile)
- * [Running Torque or MPI on Mesos](Running-torque-or-mpi-on-mesos.md)
-* [Powered-by Mesos](Powered-by-Mesos.md) -- Projects that are using Mesos!
-* [Mesos Code Internals](Mesos-Code-Internals.textile) -- an overview of the codebase and internal organization.
-* [Mesos Development Roadmap](Mesos-Roadmap.md)
+* [Powered by Mesos](Powered-by-Mesos.md) lists organizations and software that are powered by Apache Mesos.
+* [Video and Slides of Mesos Presentations](Mesos-Presentations.md)
+* Academic Papers and Project History
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/mesos/blob/5cd623e3/docs/Logging-and-Debugging.md
----------------------------------------------------------------------
diff --git a/docs/Logging-and-Debugging.md b/docs/Logging-and-Debugging.md
new file mode 100644
index 0000000..9908dca
--- /dev/null
+++ b/docs/Logging-and-Debugging.md
@@ -0,0 +1,11 @@
+---
+layout: page
+---
+
+# Logging and Debugging
+
+Mesos uses the "[Google Logging library](http://code.google.com/p/google-glog)" and writes logs to @MESOS_HOME/logs@ by default, where @MESOS_HOME@ is the location where Mesos is installed. The log directory can be "configured":Configuration.textile using the @log_dir@ parameter.
+
+Frameworks that run on Mesos have their output stored to a "work" directory on each machine. By default, this is @MESOS_HOME/work@. Within this directory, a framework's output is placed in files called @stdout@ and @stderr@ in a directory of the form @slave-X/fw-Y/Z@, where X is the slave ID, Y is the framework ID, and multiple subdirectories Z are created for each attempt to run an executor for the framework. These files can also be accessed via the web UI of the slave daemon.
+
+Finally, using an "[alpha code branch](https://github.com/mesos/mesos/tree/andyk-event-history-nowebui)" which (at the time of this writing) is still in an alpha state, structured logs of cluster activity can be written to a tab-separated text file or a SQLite database using the "event history" system.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/mesos/blob/5cd623e3/docs/Logging-and-Debugging.textile
----------------------------------------------------------------------
diff --git a/docs/Logging-and-Debugging.textile b/docs/Logging-and-Debugging.textile
deleted file mode 100644
index 0436c56..0000000
--- a/docs/Logging-and-Debugging.textile
+++ /dev/null
@@ -1,5 +0,0 @@
-Mesos uses the "Google Logging library":http://code.google.com/p/google-glog and writes logs to @MESOS_HOME/logs@ by default, where @MESOS_HOME@ is the location where Mesos is installed. The log directory can be "configured":Configuration.textile using the @log_dir@ parameter.
-
-Frameworks that run on Mesos have their output stored to a "work" directory on each machine. By default, this is @MESOS_HOME/work@. Within this directory, a framework's output is placed in files called @stdout@ and @stderr@ in a directory of the form @slave-X/fw-Y/Z@, where X is the slave ID, Y is the framework ID, and multiple subdirectories Z are created for each attempt to run an executor for the framework. These files can also be accessed via the web UI of the slave daemon.
-
-Finally, using an "alpha code branch":https://github.com/mesos/mesos/tree/andyk-event-history-nowebui which (at the time of this writing) is still in an alpha state, structured logs of cluster activity can be written to a tab-separated text file or a SQLite database using the "event history":Event-history.md system.