You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@aurora.apache.org by dl...@apache.org on 2014/10/17 06:38:27 UTC
svn commit: r1632473 - in /incubator/aurora/site:
publish/documentation/latest/configuration-reference/
publish/documentation/latest/cron-jobs/
publish/documentation/latest/images/ publish/documentation/latest/storage/
source/documentation/latest/ sour...
Author: dlester
Date: Fri Oct 17 04:38:27 2014
New Revision: 1632473
URL: http://svn.apache.org/r1632473
Log:
Updates Aurora docs to be in sync with git.
Added:
incubator/aurora/site/publish/documentation/latest/cron-jobs/
incubator/aurora/site/publish/documentation/latest/cron-jobs/index.html
incubator/aurora/site/publish/documentation/latest/images/storage_hierarchy.png (with props)
incubator/aurora/site/publish/documentation/latest/storage/
incubator/aurora/site/publish/documentation/latest/storage/index.html
incubator/aurora/site/source/documentation/latest/cron-jobs.md
incubator/aurora/site/source/documentation/latest/images/storage_hierarchy.png (with props)
incubator/aurora/site/source/documentation/latest/storage.md
Modified:
incubator/aurora/site/publish/documentation/latest/configuration-reference/index.html
incubator/aurora/site/source/documentation/latest/configuration-reference.md
Modified: incubator/aurora/site/publish/documentation/latest/configuration-reference/index.html
URL: http://svn.apache.org/viewvc/incubator/aurora/site/publish/documentation/latest/configuration-reference/index.html?rev=1632473&r1=1632472&r2=1632473&view=diff
==============================================================================
--- incubator/aurora/site/publish/documentation/latest/configuration-reference/index.html (original)
+++ incubator/aurora/site/publish/documentation/latest/configuration-reference/index.html Fri Oct 17 04:38:27 2014
@@ -527,14 +527,14 @@ resources are allocated.</p>
<td>Number of instances (sometimes referred to as replicas or shards) of the task to create. (Default: 1)</td>
</tr>
<tr>
-<td><code>cron_schedule</code> <strong>(Present, but not supported and a no-op)</strong></td>
+<td><code>cron_schedule</code></td>
<td style="text-align: center">String</td>
-<td>UTC Cron schedule in cron format. May only be used with non-service jobs. Default: None (not a cron job.)</td>
+<td>Cron schedule in cron format. May only be used with non-service jobs. See <a href="/documentation/latest/cron-jobs/">Cron Jobs</a> for more information. Default: None (not a cron job.)</td>
</tr>
<tr>
-<td><code>cron_collision_policy</code> <strong>(Present, but not supported and a no-op)</strong></td>
+<td><code>cron_collision_policy</code></td>
<td style="text-align: center">String</td>
-<td>Policy to use when a cron job is triggered while a previous run is still active. KILL<em>EXISTING Kill the previous run, and schedule the new run CANCEL</em>NEW Let the previous run continue, and cancel the new run. RUN<em>OVERLAP Let the previous run continue, and schedule the new run. (Default: KILL</em>EXISTING)</td>
+<td>Policy to use when a cron job is triggered while a previous run is still active. KILL<em>EXISTING Kill the previous run, and schedule the new run CANCEL</em>NEW Let the previous run continue, and cancel the new run. (Default: KILL_EXISTING)</td>
</tr>
<tr>
<td><code>update_config</code></td>
@@ -552,11 +552,6 @@ resources are allocated.</p>
<td>If True, restart tasks regardless of success or failure. (Default: False)</td>
</tr>
<tr>
-<td><code>daemon</code></td>
-<td style="text-align: center">Boolean</td>
-<td>A DEPRECATED alias for “service”. (Default: False)</td>
-</tr>
-<tr>
<td><code>max_task_failures</code></td>
<td style="text-align: center">Integer</td>
<td>Maximum number of failures after which the task is considered to have failed (Default: 1) Set to -1 to allow for infinite failures</td>
Added: incubator/aurora/site/publish/documentation/latest/cron-jobs/index.html
URL: http://svn.apache.org/viewvc/incubator/aurora/site/publish/documentation/latest/cron-jobs/index.html?rev=1632473&view=auto
==============================================================================
--- incubator/aurora/site/publish/documentation/latest/cron-jobs/index.html (added)
+++ incubator/aurora/site/publish/documentation/latest/cron-jobs/index.html Fri Oct 17 04:38:27 2014
@@ -0,0 +1,251 @@
+<html>
+ <head>
+ <meta charset="utf-8">
+ <title>Apache Aurora</title>
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+ <meta name="description" content="">
+ <meta name="author" content="">
+
+ <link href="/assets/css/bootstrap.css" rel="stylesheet">
+ <link href="/assets/css/bootstrap-responsive.min.css" rel="stylesheet">
+ <link href="/assets/css/main.css" rel="stylesheet">
+
+ <!-- JS -->
+ <script type="text/javascript" src="/assets/js/jquery-1.10.1.min.js"></script>
+ <script type="text/javascript" src="/assets/js/bootstrap-dropdown.js"></script>
+
+ <!-- Analytics -->
+ <script type="text/javascript">
+ var _gaq = _gaq || [];
+ _gaq.push(['_setAccount', 'UA-45879646-1']);
+ _gaq.push(['_setDomainName', 'apache.org']);
+ _gaq.push(['_trackPageview']);
+
+ (function() {
+ var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
+ ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
+ var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
+ })();
+ </script>
+ </head>
+ <body>
+ <div class="navbar navbar-static-top">
+ <div class="navbar-inner">
+ <div class="container">
+ <a href="/" class="logo"><img src="/assets/img/aurora_logo.png" alt="Apache Aurora logo" /></a>
+ <ul class="nav">
+ <li><a href="/documentation/latest/">Documentation</a></li>
+ <li><a href="/downloads/">Download</a></li>
+ <li><a href="/community">Community</a></li>
+ </ul>
+ </div>
+ </div>
+</div>
+
+<div class="container">
+<!-- magical breadcrumbs -->
+<ul class="breadcrumb">
+ <li>
+ <div class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Apache Software Foundation <b class="caret"></b></a>
+ <ul class="dropdown-menu" role="menu">
+ <li><a href="http://www.apache.org">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">Apache License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ </ul>
+ </div>
+ </li>
+ <li><span class="divider">•</span></li>
+ <li><a href="http://incubator.apache.org">Apache Incubator</a></li>
+ <li><span class="divider">•</span></li>
+ <li><a href="http://aurora.incubator.apache.org">Apache Aurora</a></li>
+</ul>
+<!-- /breadcrumb -->
+
+ <div class="container">
+ <h1 id="cron-jobs">Cron Jobs</h1>
+
+<p>Aurora supports execution of scheduled jobs on a Mesos cluster using cron-style syntax.</p>
+
+<ul>
+<li><a href="#overview">Overview</a></li>
+<li><a href="#collision-policies">Collision Policies</a>
+
+<ul>
+<li><a href="#kill_existing">KILL_EXISTING</a></li>
+<li><a href="#cancel_new">CANCEL_NEW</a></li>
+</ul></li>
+<li><a href="#failure-recovery">Failure recovery</a></li>
+<li><a href="#interacting-with-cron-jobs-via-the-aurora-cli">Interacting with cron jobs via the Aurora CLI</a>
+
+<ul>
+<li><a href="#cron-schedule">cron schedule</a></li>
+<li><a href="#cron-deschedule">cron deschedule</a></li>
+<li><a href="#cron-start">cron start</a></li>
+<li><a href="#job-killall-job-restart-job-kill">job killall, job restart, job kill</a></li>
+</ul></li>
+<li><a href="#technical-note-about-syntax">Technical Note About Syntax</a></li>
+<li><a href="#caveats">Caveats</a>
+
+<ul>
+<li><a href="#failovers">Failovers</a></li>
+<li><a href="#collision-policy-is-best-effort">Collision policy is best-effort</a></li>
+<li><a href="#timezone-configuration">Timezone Configuration</a></li>
+</ul></li>
+</ul>
+
+<h2 id="overview">Overview</h2>
+
+<p>A job is identified as a cron job by the presence of a
+<code>cron_schedule</code> attribute containing a cron-style schedule in the
+<a href="configuration-reference.md#job-objects"><code>Job</code></a> object. Examples of cron schedules
+include “every 5 minutes” (<code>*/5 * * * *</code>), “Fridays at 17:00” (<code>* 17 * * FRI</code>), and
+“the 1st and 15th day of the month at 03:00” (<code>0 3 1,15 *</code>).</p>
+
+<p>Example (available in the <a href="/documentation/latest/vagrant/">Vagrant environment</a>):</p>
+<pre class="highlight text">$ cat /vagrant/examples/job/cron_hello_world.aurora
+# cron_hello_world.aurora
+# A cron job that runs every 5 minutes.
+jobs = [
+ Job(
+ cluster = 'devcluster',
+ role = 'www-data',
+ environment = 'test',
+ name = 'cron_hello_world',
+ cron_schedule = '*/5 * * * *',
+ task = SimpleTask(
+ 'cron_hello_world',
+ 'echo "Hello world from cron, the time is now $(date --rfc-822)"'),
+ ),
+]
+</pre>
+<h2 id="collision-policies">Collision Policies</h2>
+
+<p>The <code>cron_collision_policy</code> field specifies the scheduler’s behavior when a new cron job is
+triggered while an older run hasn’t finished. The scheduler has two policies available,
+<a href="#kill_existing">KILL_EXISTING</a> and <a href="#cancel_new">CANCEL_NEW</a>.</p>
+
+<h3 id="kill_existing">KILL_EXISTING</h3>
+
+<p>The default policy - on a collision the old instances are killed and a instances with the current
+configuration are started.</p>
+
+<h3 id="cancel_new">CANCEL_NEW</h3>
+
+<p>On a collision the new run is cancelled.</p>
+
+<p>Note that the use of this flag is likely a code smell - interrupted cron jobs should be able
+to recover their progress on a subsequent invocation, otherwise they risk having their work queue
+grow faster than they can process it.</p>
+
+<h2 id="failure-recovery">Failure recovery</h2>
+
+<p>Unlike with services, which aurora will always re-execute regardless of exit status, instances of
+cron jobs retry according to the <code>max_task_failures</code> attribute of the
+<a href="configuration-reference.md#task-objects">Task</a> object. To get “run-until-failure” semantics,
+set <code>max_task_failures</code> to <code>-1</code>.</p>
+
+<h2 id="interacting-with-cron-jobs-via-the-aurora-cli">Interacting with cron jobs via the Aurora CLI</h2>
+
+<p>Most interaction with cron jobs takes place using the <code>cron</code> subcommand. See <code>aurora2 help cron</code>
+for up-to-date usage instructions.</p>
+
+<h3 id="cron-schedule">cron schedule</h3>
+
+<p>Schedules a new cron job on the Aurora cluster for later runs, or updates an existing job.</p>
+<pre class="highlight text">$ aurora2 cron schedule devcluster/www-data/test/cron_hello_world /vagrant/examples/jobs/cron_hello_world.aurora
+</pre>
+<h3 id="cron-deschedule">cron deschedule</h3>
+
+<p>Deschedules a cron job, preventing future runs but allowing current runs to complete.</p>
+<pre class="highlight text">$ aurora2 cron deschedule devcluster/www-data/test/cron_hello_world
+</pre>
+<h3 id="cron-start">cron start</h3>
+
+<p>Start a cron job immediately, outside of its normal cron schedule.</p>
+<pre class="highlight text">$ aurora2 cron start devcluster/www-data/test/cron_hello_world
+</pre>
+<h3 id="job-killall,-job-restart,-job-kill">job killall, job restart, job kill</h3>
+
+<p>Cron jobs create instances running on the cluster that you can interact with like normal Aurora
+tasks with <code>job kill</code> and <code>job restart</code>.</p>
+
+<h2 id="technical-note-about-syntax">Technical Note About Syntax</h2>
+
+<p><code>cron_schedule</code> uses a restricted subset of BSD crontab syntax. While the
+execution engine currently uses Quartz, the schedule parsing is custom, a subset of FreeBSD
+<a href="http://www.freebsd.org/cgi/man.cgi?crontab(5)">crontab(5)</a> syntax. See
+<a href="https://github.com/apache/incubator-aurora/blob/master/src/main/java/org/apache/aurora/scheduler/cron/CrontabEntry.java#L106-L124">the source</a>
+for details.</p>
+
+<h2 id="caveats">Caveats</h2>
+
+<h3 id="failovers">Failovers</h3>
+
+<p>No failover recovery. Aurora does not record the latest minute it fired
+triggers for across failovers. Therefore it’s possible to miss triggers
+on failover. Note that this behavior may change in the future.</p>
+
+<p>It’s necessary to sync time between schedulers with something like <code>ntpd</code>.
+Clock skew could cause double or missed triggers in the case of a failover.</p>
+
+<h3 id="collision-policy-is-best-effort">Collision policy is best-effort</h3>
+
+<p>Aurora aims to always have <em>at least one copy</em> of a given instance running at a time - it’s
+an AP system, meaning it chooses Availability and Partition Tolerance at the expense of
+Consistency.</p>
+
+<p>If your collision policy was <code>CANCEL_NEW</code> and a task has terminated but
+Aurora has not noticed this Aurora will go ahead and create your new
+task.</p>
+
+<p>If your collision policy was <code>KILL_EXISTING</code> and a task was marked <code>LOST</code>
+but not yet GCed Aurora will go ahead and create your new task without
+attempting to kill the old one (outside the GC interval).</p>
+
+<h3 id="timezone-configuration">Timezone Configuration</h3>
+
+<p>Cron timezone is configured indepdendently of JVM timezone with the <code>-cron_timezone</code> flag and
+defaults to UTC.</p>
+
+ </div>
+ <div class="container">
+ <hr>
+ <footer class="footer">
+ <div class="row-fluid">
+ <div class="span2 text-left">
+ <h3>Links</h3>
+ <ul class="unstyled">
+ <li><a href="/downloads/">Downloads</a></li>
+ <li><a href="/developers/">Developers</a></li>
+ </ul>
+ </div>
+ <div class="span3 text-left">
+ <h3>Community</h3>
+ <ul class="unstyled">
+ <li><a href="/community/">Mailing Lists</a></li>
+ <li><a href="http://issues.apache.org/jira/browse/aurora">Issue Tracking</a></li>
+ <li><a href="/docs/howtocontribute/">How To Contribute</a></li>
+ </ul>
+ </div>
+ <div class="span7 text-left">
+ <h3>Apache Software Foundation</h3>
+
+ <div class="span8">
+ Copyright 2014 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. Apache, Apache Thrift, and the Apache feather logo are trademarks of The Apache Software Foundation. Currently part of the <a href="http://incubator.apache.org">Apache Incubator</a>.
+ </div>
+ <div class=" pull-right">
+ <a href="http://incubator.apache.org" class="logo"><img src="/assets/img/apache_incubator_logo.png" alt="Apache Incubator" class="pull-right"/></a>
+ </div>
+ </div>
+
+ </div>
+
+ </footer>
+</div>
+
+ </body>
+</html>
+
Added: incubator/aurora/site/publish/documentation/latest/images/storage_hierarchy.png
URL: http://svn.apache.org/viewvc/incubator/aurora/site/publish/documentation/latest/images/storage_hierarchy.png?rev=1632473&view=auto
==============================================================================
Binary file - no diff available.
Propchange: incubator/aurora/site/publish/documentation/latest/images/storage_hierarchy.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: incubator/aurora/site/publish/documentation/latest/storage/index.html
URL: http://svn.apache.org/viewvc/incubator/aurora/site/publish/documentation/latest/storage/index.html?rev=1632473&view=auto
==============================================================================
--- incubator/aurora/site/publish/documentation/latest/storage/index.html (added)
+++ incubator/aurora/site/publish/documentation/latest/storage/index.html Fri Oct 17 04:38:27 2014
@@ -0,0 +1,202 @@
+<html>
+ <head>
+ <meta charset="utf-8">
+ <title>Apache Aurora</title>
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+ <meta name="description" content="">
+ <meta name="author" content="">
+
+ <link href="/assets/css/bootstrap.css" rel="stylesheet">
+ <link href="/assets/css/bootstrap-responsive.min.css" rel="stylesheet">
+ <link href="/assets/css/main.css" rel="stylesheet">
+
+ <!-- JS -->
+ <script type="text/javascript" src="/assets/js/jquery-1.10.1.min.js"></script>
+ <script type="text/javascript" src="/assets/js/bootstrap-dropdown.js"></script>
+
+ <!-- Analytics -->
+ <script type="text/javascript">
+ var _gaq = _gaq || [];
+ _gaq.push(['_setAccount', 'UA-45879646-1']);
+ _gaq.push(['_setDomainName', 'apache.org']);
+ _gaq.push(['_trackPageview']);
+
+ (function() {
+ var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
+ ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
+ var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
+ })();
+ </script>
+ </head>
+ <body>
+ <div class="navbar navbar-static-top">
+ <div class="navbar-inner">
+ <div class="container">
+ <a href="/" class="logo"><img src="/assets/img/aurora_logo.png" alt="Apache Aurora logo" /></a>
+ <ul class="nav">
+ <li><a href="/documentation/latest/">Documentation</a></li>
+ <li><a href="/downloads/">Download</a></li>
+ <li><a href="/community">Community</a></li>
+ </ul>
+ </div>
+ </div>
+</div>
+
+<div class="container">
+<!-- magical breadcrumbs -->
+<ul class="breadcrumb">
+ <li>
+ <div class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Apache Software Foundation <b class="caret"></b></a>
+ <ul class="dropdown-menu" role="menu">
+ <li><a href="http://www.apache.org">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">Apache License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ </ul>
+ </div>
+ </li>
+ <li><span class="divider">•</span></li>
+ <li><a href="http://incubator.apache.org">Apache Incubator</a></li>
+ <li><span class="divider">•</span></li>
+ <li><a href="http://aurora.incubator.apache.org">Apache Aurora</a></li>
+</ul>
+<!-- /breadcrumb -->
+
+ <div class="container">
+ <h1 id="aurora-scheduler-storage">Aurora Scheduler Storage</h1>
+
+<ul>
+<li><a href="#overview">Overview</a></li>
+<li><a href="#reads-writes-modifications">Reads, writes, modifications…</a>
+
+<ul>
+<li><a href="#read-lifecycle">Read lifecycle</a></li>
+<li><a href="#write-lifecycle">Write lifecycle</a></li>
+</ul></li>
+<li><a href="#atomicity-consistency-and-isolation">Atomicity, consistency and isolation</a></li>
+<li><a href="#population-on-restart">Population on restart</a></li>
+</ul>
+
+<h2 id="overview">Overview</h2>
+
+<p>Aurora scheduler maintains data that need to be persisted to survive failovers and restarts.
+For example:</p>
+
+<ul>
+<li>Task configurations and scheduled task instances</li>
+<li>Job update configurations and update progress</li>
+<li>Production resource quotas</li>
+<li>Mesos resource offer host attributes</li>
+</ul>
+
+<p>Aurora solves its persistence needs by leveraging the Mesos implementation of a Paxos replicated
+log <a href="https://ramcloud.stanford.edu/~ongaro/userstudy/paxos.pdf">[1]</a>
+<a href="http://en.wikipedia.org/wiki/State_machine_replication">[2]</a> with a key-value
+<a href="https://github.com/google/leveldb">LevelDB</a> storage as persistence media.</p>
+
+<p>Conceptually, it can be represented by the following major components:</p>
+
+<ul>
+<li>Volatile storage: in-memory cache of all available data. Implemented via in-memory
+<a href="http://www.h2database.com/html/main.html">H2 Database</a> and accessed via
+<a href="http://mybatis.github.io/mybatis-3/">MyBatis</a>.</li>
+<li>Log manager: interface between Aurora storage and Mesos replicated log. The default schema format
+is <a href="https://github.com/apache/thrift">thrift</a>. Data is stored in serialized binary form.</li>
+<li>Snapshot manager: all data is periodically persisted in Mesos replicated log in a single snapshot.
+This helps establishing periodic recovery checkpoints and speeds up volatile storage recovery on
+restart.</li>
+<li>Backup manager: as a precaution, snapshots are periodically written out into backup files.
+This solves a disaster recovery problem in case of a complete loss or corruption of Mesos log files.</li>
+</ul>
+
+<p><img alt="Storage hierarchy" src="../images/storage_hierarchy.png" /></p>
+
+<h2 id="reads,-writes,-modifications...">Reads, writes, modifications…</h2>
+
+<p>All services in Aurora access data via a set of predefined store interfaces (aka stores) logically
+grouped by the type of data they serve. Every interface defines a specific set of operations allowed
+on the data thus abstracting out the storage access and the actual persistence implementation. The
+latter is especially important in view of a general immutability of persisted data. With the Mesos
+replicated log as the underlying persistence solution, data can be read and written easily but not
+modified. All modifications are simulated by saving new versions of modified objects. This feature
+and general performance considerations justify the existence of the volatile in-memory store.</p>
+
+<h3 id="read-lifecycle">Read lifecycle</h3>
+
+<p>There are two types of reads available in Aurora: consistent and weakly-consistent. The difference
+is explained <a href="#atomicity-and-isolation">below</a>.</p>
+
+<p>All reads are served from the volatile storage making reads generally cheap storage operations
+from the performance standpoint. The majority of the volatile stores are represented by the
+in-memory H2 database. This allows for rich schema definitions, queries and relationships that
+key-value storage is unable to match.</p>
+
+<h3 id="write-lifecycle">Write lifecycle</h3>
+
+<p>Writes are more involved operations since in addition to updating the volatile store data has to be
+appended to the replicated log. Data is not available for reads until fully ack-ed by both
+replicated log and volatile storage.</p>
+
+<h2 id="atomicity,-consistency-and-isolation">Atomicity, consistency and isolation</h2>
+
+<p>Aurora uses <a href="http://en.wikipedia.org/wiki/Write-ahead_logging">write-ahead logging</a> to ensure
+consistency between replicated and volatile storage. In Aurora, data is first written into the
+replicated log and only then updated in the volatile store.</p>
+
+<p>Aurora storage uses read-write locks to serialize data mutations and provide consistent view of the
+available data. The available <code>Storage</code> interface exposes 3 major types of operations:
+* <code>consistentRead</code> - access is locked using reader’s lock and provides consistent view on read
+* <code>weaklyConsistentRead</code> - access is lock-less. Delivers best contention performance but may result
+in stale reads
+* <code>write</code> - access is fully serialized by using writer’s lock. Operation success requires both
+volatile and replicated writes to succeed.</p>
+
+<p>The consistency of the volatile store is enforced via H2 transactional isolation.</p>
+
+<h2 id="population-on-restart">Population on restart</h2>
+
+<p>Any time a scheduler restarts, it restores its volatile state from the most recent position recorded
+in the replicated log by restoring the snapshot and replaying individual log entries on top to fully
+recover the state up to the last write.</p>
+
+ </div>
+ <div class="container">
+ <hr>
+ <footer class="footer">
+ <div class="row-fluid">
+ <div class="span2 text-left">
+ <h3>Links</h3>
+ <ul class="unstyled">
+ <li><a href="/downloads/">Downloads</a></li>
+ <li><a href="/developers/">Developers</a></li>
+ </ul>
+ </div>
+ <div class="span3 text-left">
+ <h3>Community</h3>
+ <ul class="unstyled">
+ <li><a href="/community/">Mailing Lists</a></li>
+ <li><a href="http://issues.apache.org/jira/browse/aurora">Issue Tracking</a></li>
+ <li><a href="/docs/howtocontribute/">How To Contribute</a></li>
+ </ul>
+ </div>
+ <div class="span7 text-left">
+ <h3>Apache Software Foundation</h3>
+
+ <div class="span8">
+ Copyright 2014 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. Apache, Apache Thrift, and the Apache feather logo are trademarks of The Apache Software Foundation. Currently part of the <a href="http://incubator.apache.org">Apache Incubator</a>.
+ </div>
+ <div class=" pull-right">
+ <a href="http://incubator.apache.org" class="logo"><img src="/assets/img/apache_incubator_logo.png" alt="Apache Incubator" class="pull-right"/></a>
+ </div>
+ </div>
+
+ </div>
+
+ </footer>
+</div>
+
+ </body>
+</html>
+
Modified: incubator/aurora/site/source/documentation/latest/configuration-reference.md
URL: http://svn.apache.org/viewvc/incubator/aurora/site/source/documentation/latest/configuration-reference.md?rev=1632473&r1=1632472&r2=1632473&view=diff
==============================================================================
--- incubator/aurora/site/source/documentation/latest/configuration-reference.md (original)
+++ incubator/aurora/site/source/documentation/latest/configuration-reference.md Fri Oct 17 04:38:27 2014
@@ -313,12 +313,11 @@ Job Schema
```environment``` | String | Job environment, default ```devel```. Must be one of ```prod```, ```devel```, ```test``` or ```staging<number>```.
```contact``` | String | Best email address to reach the owner of the job. For production jobs, this is usually a team mailing list.
```instances```| Integer | Number of instances (sometimes referred to as replicas or shards) of the task to create. (Default: 1)
- ```cron_schedule``` **(Present, but not supported and a no-op)** | String | UTC Cron schedule in cron format. May only be used with non-service jobs. Default: None (not a cron job.)
- ```cron_collision_policy``` **(Present, but not supported and a no-op)** | String | Policy to use when a cron job is triggered while a previous run is still active. KILL_EXISTING Kill the previous run, and schedule the new run CANCEL_NEW Let the previous run continue, and cancel the new run. RUN_OVERLAP Let the previous run continue, and schedule the new run. (Default: KILL_EXISTING)
+ ```cron_schedule``` | String | Cron schedule in cron format. May only be used with non-service jobs. See [Cron Jobs](/documentation/latest/cron-jobs/) for more information. Default: None (not a cron job.)
+ ```cron_collision_policy``` | String | Policy to use when a cron job is triggered while a previous run is still active. KILL_EXISTING Kill the previous run, and schedule the new run CANCEL_NEW Let the previous run continue, and cancel the new run. (Default: KILL_EXISTING)
```update_config``` | ```update_config``` object | Parameters for controlling the rate and policy of rolling updates.
```constraints``` | dict | Scheduling constraints for the tasks. See the section on the [constraint specification language](#Specifying-Scheduling-Constraints)
```service``` | Boolean | If True, restart tasks regardless of success or failure. (Default: False)
- ```daemon``` | Boolean | A DEPRECATED alias for "service". (Default: False)
```max_task_failures``` | Integer | Maximum number of failures after which the task is considered to have failed (Default: 1) Set to -1 to allow for infinite failures
```priority``` | Integer | Preemption priority to give the task (Default 0). Tasks with higher priorities may preempt tasks at lower priorities.
```production``` | Boolean | Whether or not this is a production task backed by quota (Default: False). Production jobs may preempt any non-production job, and may only be preempted by production jobs in the same role and of higher priority. To run jobs at this level, the job role must have the appropriate quota.
Added: incubator/aurora/site/source/documentation/latest/cron-jobs.md
URL: http://svn.apache.org/viewvc/incubator/aurora/site/source/documentation/latest/cron-jobs.md?rev=1632473&view=auto
==============================================================================
--- incubator/aurora/site/source/documentation/latest/cron-jobs.md (added)
+++ incubator/aurora/site/source/documentation/latest/cron-jobs.md Fri Oct 17 04:38:27 2014
@@ -0,0 +1,130 @@
+# Cron Jobs
+
+Aurora supports execution of scheduled jobs on a Mesos cluster using cron-style syntax.
+
+- [Overview](#overview)
+- [Collision Policies](#collision-policies)
+ - [KILL_EXISTING](#kill_existing)
+ - [CANCEL_NEW](#cancel_new)
+- [Failure recovery](#failure-recovery)
+- [Interacting with cron jobs via the Aurora CLI](#interacting-with-cron-jobs-via-the-aurora-cli)
+ - [cron schedule](#cron-schedule)
+ - [cron deschedule](#cron-deschedule)
+ - [cron start](#cron-start)
+ - [job killall, job restart, job kill](#job-killall-job-restart-job-kill)
+- [Technical Note About Syntax](#technical-note-about-syntax)
+- [Caveats](#caveats)
+ - [Failovers](#failovers)
+ - [Collision policy is best-effort](#collision-policy-is-best-effort)
+ - [Timezone Configuration](#timezone-configuration)
+
+## Overview
+
+A job is identified as a cron job by the presence of a
+`cron_schedule` attribute containing a cron-style schedule in the
+[`Job`](configuration-reference.md#job-objects) object. Examples of cron schedules
+include "every 5 minutes" (`*/5 * * * *`), "Fridays at 17:00" (`* 17 * * FRI`), and
+"the 1st and 15th day of the month at 03:00" (`0 3 1,15 *`).
+
+Example (available in the [Vagrant environment](/documentation/latest/vagrant/)):
+
+ $ cat /vagrant/examples/job/cron_hello_world.aurora
+ # cron_hello_world.aurora
+ # A cron job that runs every 5 minutes.
+ jobs = [
+ Job(
+ cluster = 'devcluster',
+ role = 'www-data',
+ environment = 'test',
+ name = 'cron_hello_world',
+ cron_schedule = '*/5 * * * *',
+ task = SimpleTask(
+ 'cron_hello_world',
+ 'echo "Hello world from cron, the time is now $(date --rfc-822)"'),
+ ),
+ ]
+
+## Collision Policies
+
+The `cron_collision_policy` field specifies the scheduler's behavior when a new cron job is
+triggered while an older run hasn't finished. The scheduler has two policies available,
+[KILL_EXISTING](#kill_existing) and [CANCEL_NEW](#cancel_new).
+
+### KILL_EXISTING
+
+The default policy - on a collision the old instances are killed and a instances with the current
+configuration are started.
+
+### CANCEL_NEW
+
+On a collision the new run is cancelled.
+
+Note that the use of this flag is likely a code smell - interrupted cron jobs should be able
+to recover their progress on a subsequent invocation, otherwise they risk having their work queue
+grow faster than they can process it.
+
+## Failure recovery
+
+Unlike with services, which aurora will always re-execute regardless of exit status, instances of
+cron jobs retry according to the `max_task_failures` attribute of the
+[Task](configuration-reference.md#task-objects) object. To get "run-until-failure" semantics,
+set `max_task_failures` to `-1`.
+
+## Interacting with cron jobs via the Aurora CLI
+
+Most interaction with cron jobs takes place using the `cron` subcommand. See `aurora2 help cron`
+for up-to-date usage instructions.
+
+### cron schedule
+Schedules a new cron job on the Aurora cluster for later runs, or updates an existing job.
+
+ $ aurora2 cron schedule devcluster/www-data/test/cron_hello_world /vagrant/examples/jobs/cron_hello_world.aurora
+
+### cron deschedule
+Deschedules a cron job, preventing future runs but allowing current runs to complete.
+
+ $ aurora2 cron deschedule devcluster/www-data/test/cron_hello_world
+
+### cron start
+Start a cron job immediately, outside of its normal cron schedule.
+
+ $ aurora2 cron start devcluster/www-data/test/cron_hello_world
+
+### job killall, job restart, job kill
+Cron jobs create instances running on the cluster that you can interact with like normal Aurora
+tasks with `job kill` and `job restart`.
+
+## Technical Note About Syntax
+
+`cron_schedule` uses a restricted subset of BSD crontab syntax. While the
+execution engine currently uses Quartz, the schedule parsing is custom, a subset of FreeBSD
+[crontab(5)](http://www.freebsd.org/cgi/man.cgi?crontab(5)) syntax. See
+[the source](https://github.com/apache/incubator-aurora/blob/master/src/main/java/org/apache/aurora/scheduler/cron/CrontabEntry.java#L106-L124)
+for details.
+
+## Caveats
+
+### Failovers
+No failover recovery. Aurora does not record the latest minute it fired
+triggers for across failovers. Therefore it's possible to miss triggers
+on failover. Note that this behavior may change in the future.
+
+It's necessary to sync time between schedulers with something like `ntpd`.
+Clock skew could cause double or missed triggers in the case of a failover.
+
+### Collision policy is best-effort
+Aurora aims to always have *at least one copy* of a given instance running at a time - it's
+an AP system, meaning it chooses Availability and Partition Tolerance at the expense of
+Consistency.
+
+If your collision policy was `CANCEL_NEW` and a task has terminated but
+Aurora has not noticed this Aurora will go ahead and create your new
+task.
+
+If your collision policy was `KILL_EXISTING` and a task was marked `LOST`
+but not yet GCed Aurora will go ahead and create your new task without
+attempting to kill the old one (outside the GC interval).
+
+### Timezone Configuration
+Cron timezone is configured indepdendently of JVM timezone with the `-cron_timezone` flag and
+defaults to UTC.
Added: incubator/aurora/site/source/documentation/latest/images/storage_hierarchy.png
URL: http://svn.apache.org/viewvc/incubator/aurora/site/source/documentation/latest/images/storage_hierarchy.png?rev=1632473&view=auto
==============================================================================
Binary file - no diff available.
Propchange: incubator/aurora/site/source/documentation/latest/images/storage_hierarchy.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: incubator/aurora/site/source/documentation/latest/storage.md
URL: http://svn.apache.org/viewvc/incubator/aurora/site/source/documentation/latest/storage.md?rev=1632473&view=auto
==============================================================================
--- incubator/aurora/site/source/documentation/latest/storage.md (added)
+++ incubator/aurora/site/source/documentation/latest/storage.md Fri Oct 17 04:38:27 2014
@@ -0,0 +1,87 @@
+#Aurora Scheduler Storage
+
+- [Overview](#overview)
+- [Reads, writes, modifications...](#reads-writes-modifications)
+ - [Read lifecycle](#read-lifecycle)
+ - [Write lifecycle](#write-lifecycle)
+- [Atomicity, consistency and isolation](#atomicity-consistency-and-isolation)
+- [Population on restart](#population-on-restart)
+
+## Overview
+
+Aurora scheduler maintains data that need to be persisted to survive failovers and restarts.
+For example:
+
+* Task configurations and scheduled task instances
+* Job update configurations and update progress
+* Production resource quotas
+* Mesos resource offer host attributes
+
+Aurora solves its persistence needs by leveraging the Mesos implementation of a Paxos replicated
+log [[1]](https://ramcloud.stanford.edu/~ongaro/userstudy/paxos.pdf)
+[[2]](http://en.wikipedia.org/wiki/State_machine_replication) with a key-value
+[LevelDB](https://github.com/google/leveldb) storage as persistence media.
+
+Conceptually, it can be represented by the following major components:
+
+* Volatile storage: in-memory cache of all available data. Implemented via in-memory
+[H2 Database](http://www.h2database.com/html/main.html) and accessed via
+[MyBatis](http://mybatis.github.io/mybatis-3/).
+* Log manager: interface between Aurora storage and Mesos replicated log. The default schema format
+is [thrift](https://github.com/apache/thrift). Data is stored in serialized binary form.
+* Snapshot manager: all data is periodically persisted in Mesos replicated log in a single snapshot.
+This helps establishing periodic recovery checkpoints and speeds up volatile storage recovery on
+restart.
+* Backup manager: as a precaution, snapshots are periodically written out into backup files.
+This solves a disaster recovery problem in case of a complete loss or corruption of Mesos log files.
+
+
+
+## Reads, writes, modifications...
+
+All services in Aurora access data via a set of predefined store interfaces (aka stores) logically
+grouped by the type of data they serve. Every interface defines a specific set of operations allowed
+on the data thus abstracting out the storage access and the actual persistence implementation. The
+latter is especially important in view of a general immutability of persisted data. With the Mesos
+replicated log as the underlying persistence solution, data can be read and written easily but not
+modified. All modifications are simulated by saving new versions of modified objects. This feature
+and general performance considerations justify the existence of the volatile in-memory store.
+
+### Read lifecycle
+
+There are two types of reads available in Aurora: consistent and weakly-consistent. The difference
+is explained [below](#atomicity-and-isolation).
+
+All reads are served from the volatile storage making reads generally cheap storage operations
+from the performance standpoint. The majority of the volatile stores are represented by the
+in-memory H2 database. This allows for rich schema definitions, queries and relationships that
+key-value storage is unable to match.
+
+### Write lifecycle
+
+Writes are more involved operations since in addition to updating the volatile store data has to be
+appended to the replicated log. Data is not available for reads until fully ack-ed by both
+replicated log and volatile storage.
+
+## Atomicity, consistency and isolation
+
+Aurora uses [write-ahead logging](http://en.wikipedia.org/wiki/Write-ahead_logging) to ensure
+consistency between replicated and volatile storage. In Aurora, data is first written into the
+replicated log and only then updated in the volatile store.
+
+Aurora storage uses read-write locks to serialize data mutations and provide consistent view of the
+available data. The available `Storage` interface exposes 3 major types of operations:
+* `consistentRead` - access is locked using reader's lock and provides consistent view on read
+* `weaklyConsistentRead` - access is lock-less. Delivers best contention performance but may result
+in stale reads
+* `write` - access is fully serialized by using writer's lock. Operation success requires both
+volatile and replicated writes to succeed.
+
+The consistency of the volatile store is enforced via H2 transactional isolation.
+
+## Population on restart
+
+Any time a scheduler restarts, it restores its volatile state from the most recent position recorded
+in the replicated log by restoring the snapshot and replaying individual log entries on top to fully
+recover the state up to the last write.
+