svn commit: r971096 [2/8] - in /websites/staging/slider/trunk/content: ./ design/ design/registry/ design/specification/ developing/ docs/ docs/api/ docs/configuration/ docs/configuration/revision-1/ docs/slider_specs/ downloads/ release_notes/

[bu...@apache.org]

Modified: websites/staging/slider/trunk/content/design/rolehistory.html
==============================================================================
--- websites/staging/slider/trunk/content/design/rolehistory.html (original)
+++ websites/staging/slider/trunk/content/design/rolehistory.html Mon Nov  2 17:24:47 2015
@@ -155,7 +155,7 @@
   <div style="text-align: center">
     <h1><a href="/index.html">Apache Slider (incubating)</a></h1>
     <hr>
-Latest release: <strong>0.80.0-incubating</strong><br>
+Latest release: <strong>0.81.1-incubating</strong><br>
     <br>
     <a id="download-button-sidebar" class="btn btn-success btn-block" href="/downloads/" role="button">Download</a>
   </div>
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
 
     <h1 class="title"></h1>
 
-    <!---
+    <style type="text/css">
+/* The following code is added by mdx_elementid.py
+   It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+  visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink, h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink, dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
    Licensed to the Apache Software Foundation (ASF) under one or more
    contributor license agreements.  See the NOTICE file distributed with
    this work for additional information regarding copyright ownership.
@@ -185,10 +196,10 @@ Latest release: <strong>0.80.0-incubatin
    limitations under the License.
 -->
 
-<h1 id="apache-slider-placement-how-slider-brings-back-nodes-in-the-same-location">Apache Slider Placement: how Slider brings back nodes in the same location</h1>
-<h3 id="last-updated-2015-03-19">Last updated  2015-03-19</h3>
-<h2 id="changes">Changes</h2>
-<h3 id="slider-080-incubating">Slider 0.80-incubating</h3>
+<h1 id="apache-slider-placement-how-slider-brings-back-nodes-in-the-same-location">Apache Slider Placement: how Slider brings back nodes in the same location<a class="headerlink" href="#apache-slider-placement-how-slider-brings-back-nodes-in-the-same-location" title="Permanent link">&para;</a></h1>
+<h3 id="last-updated-2015-03-19">Last updated  2015-03-19<a class="headerlink" href="#last-updated-2015-03-19" title="Permanent link">&para;</a></h3>
+<h2 id="changes">Changes<a class="headerlink" href="#changes" title="Permanent link">&para;</a></h2>
+<h3 id="slider-080-incubating">Slider 0.80-incubating<a class="headerlink" href="#slider-080-incubating" title="Permanent link">&para;</a></h3>
 <p>A major rework of placement has taken place, <a href="https://issues.apache.org/jira/browse/SLIDER-611">Über-JIRA : placement phase 2</a></p>
 <ol>
 <li>Slider manages the process of relaxing a request from a specific host to "anywhere in the cluster".</li>
@@ -216,7 +227,7 @@ exit.</li>
 </ul>
 </li>
 </ol>
-<h5 id="role-history-reloading-enhancements">Role History Reloading Enhancements</h5>
+<h5 id="role-history-reloading-enhancements">Role History Reloading Enhancements<a class="headerlink" href="#role-history-reloading-enhancements" title="Permanent link">&para;</a></h5>
 <p>How persisted role history has also been improved [SLIDER-600]((https://issues.apache.org/jira/browse/SLIDER-600)</p>
 <ol>
 <li>Reloading of persistent history has been made resilient to changes in the number of roles.</li>
@@ -240,7 +251,7 @@ reloading entries there are two tactics<
 <li>Delete the JSON files</li>
 <li>Edit the most recent JSON file and delete any entry with the type <code>RoleHistoryMapping</code>.</li>
 </ol>
-<h4 id="placement-policies">Placement policies</h4>
+<h4 id="placement-policies">Placement policies<a class="headerlink" href="#placement-policies" title="Permanent link">&para;</a></h4>
 <p><code>strict</code> placement</p>
 <p>Again, "strict placement" has a different policy: once a component has been deployed on a node,
 one component request will be made against that node, even if it is considered unreliable. No
@@ -252,7 +263,7 @@ While tracking of recent failure counts
 <p>There's still no explicit support for this in YARN or slider. As noted above, Slider does
 try to spread placement when rebuilding an application, but otherwise it accepts which
 hosts YARN allocates containers on.</p>
-<h3 id="slider-070-incubating">Slider 0.70-incubating</h3>
+<h3 id="slider-070-incubating">Slider 0.70-incubating<a class="headerlink" href="#slider-070-incubating" title="Permanent link">&para;</a></h3>
 <p>The Slider AM tracks the recent failure count of each role on each (known) YARN node; when
 the failure count of a role is above a role-specific threshold, no more explicit requests are
 made for that host. Exception: "strict placement" component deployments are always requsted
@@ -261,7 +272,7 @@ against the node irrespective of their h
 Slider's placement history indefinitely. The fact that a node was unreliable two weeks ago should
 not mean that it should be blacklisted this week. </p>
 <p>Note that this failure history is not persisted; it is lost on AM start/restart.</p>
-<h3 id="slider-060-incubating">Slider 0.60-incubating</h3>
+<h3 id="slider-060-incubating">Slider 0.60-incubating<a class="headerlink" href="#slider-060-incubating" title="Permanent link">&para;</a></h3>
 <ol>
 <li>Slider supports <code>strict placement</code> in which a request is when made for a previously
 used node and never used again.</li>
@@ -272,7 +283,7 @@ nodes. As such, it offers significant co
 hardware to specific applications. Note that the HDFS filesystem is still shared across the cluster,
 so IO from other in-cluster applications will impinge on each other.</li>
 </ol>
-<h2 id="introduction">Introduction</h2>
+<h2 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">&para;</a></h2>
 <p>Slider needs to bring up instances of a given role on the machine(s) on which
 they last ran. It must remember after shrinking or freezing an Application Instance  which
 servers were last used for a role. It must then try to use this (persisted) data to select
@@ -289,7 +300,7 @@ tables.</p>
 <p>Note that it does not need to care about the placement of other roles, such
 as the HBase masters -there anti-affinity between other instances is
 the key requirement.</p>
-<h3 id="terminology">Terminology</h3>
+<h3 id="terminology">Terminology<a class="headerlink" href="#terminology" title="Permanent link">&para;</a></h3>
 <ul>
 <li><strong>Node</strong> : A server in the YARN Physical (or potentially virtual) cluster of servers.</li>
 <li><strong>Application Instance</strong>: A deployment of a distributed application as created and
@@ -304,7 +315,7 @@ and so retained across much of the code
 manage its Application Instance.</li>
 <li><strong>RM</strong> YARN Resource Manager</li>
 </ul>
-<h3 id="assumptions">Assumptions</h3>
+<h3 id="assumptions">Assumptions<a class="headerlink" href="#assumptions" title="Permanent link">&para;</a></h3>
 <p>Here are some assumptions in Slider's design</p>
 <ol>
 <li>
@@ -360,7 +371,7 @@ designing the RoleHistory datastructures
 of rolling statistics on recent failures would be a first step to this </p>
 </li>
 </ol>
-<h3 id="the-rolehistory-datastructure">The RoleHistory Datastructure</h3>
+<h3 id="the-rolehistory-datastructure">The RoleHistory Datastructure<a class="headerlink" href="#the-rolehistory-datastructure" title="Permanent link">&para;</a></h3>
 <p>The <code>RoleHistory</code> is a datastructure which models the component assignment, and
 can persist it to and restore it from the (shared) filesystem.</p>
 <ul>
@@ -409,7 +420,7 @@ outstanding request - are considered can
 of the time-ordered list of nodes that last ran an instance of that component</p>
 </li>
 </ul>
-<h3 id="persistence">Persistence</h3>
+<h3 id="persistence">Persistence<a class="headerlink" href="#persistence" title="Permanent link">&para;</a></h3>
 <p>The state of the component is persisted to HDFS on changes —but not on Application Instance
 termination.</p>
 <ol>
@@ -433,7 +444,7 @@ might seem the approach most consistent
 an easy structure to work with.</p>
 <p>The initial implementation will use Apache Avro as the persistence format,
 with the data saved in JSON or compressed format.</p>
-<h2 id="weaknesses-in-this-design">Weaknesses in this design</h2>
+<h2 id="weaknesses-in-this-design">Weaknesses in this design<a class="headerlink" href="#weaknesses-in-this-design" title="Permanent link">&para;</a></h2>
 <p><strong>Blacklisting</strong>: even if a node fails repeatedly, this design will still try to re-request
 instances on this node; there is no blacklisting. As a central blacklist
 for YARN has been proposed, it is hoped that this issue will be addressed centrally,
@@ -452,7 +463,7 @@ history of a node -maintaining some kind
 node use and picking the heaviest used, or some other more-complex algorithm.
 This may be possible, but we'd need evidence that the problem existed before
 trying to address it.</p>
-<h1 id="the-nodemap">The NodeMap</h1>
+<h1 id="the-nodemap">The NodeMap<a class="headerlink" href="#the-nodemap" title="Permanent link">&para;</a></h1>
 <p>A core data structure, the <code>NodeMap</code>, is a map of every known node in the YARN cluster, tracking
 how many containers are allocated to specific roles in it, and, when there
 are no active instances, when it was last used. This history is used to
@@ -475,7 +486,7 @@ most containers on it. This would spread
 <li>When there are no empty nodes to request containers on, a request would
 let YARN choose.</li>
 </ol>
-<h4 id="strengths">Strengths</h4>
+<h4 id="strengths">Strengths<a class="headerlink" href="#strengths" title="Permanent link">&para;</a></h4>
 <ul>
 <li>Handles the multi-container on one node problem</li>
 <li>By storing details about every role, cross-role decisions could be possible</li>
@@ -485,7 +496,7 @@ let YARN choose.</li>
 <li>Easy to view and debug</li>
 <li>Would support cross-role collection of node failures in future</li>
 </ul>
-<h4 id="weaknesses">Weaknesses</h4>
+<h4 id="weaknesses">Weaknesses<a class="headerlink" href="#weaknesses" title="Permanent link">&para;</a></h4>
 <ul>
 <li>Size of the data structure is <code>O(nodes * role-instances</code>). This
 could be mitigated by regular cleansing of the structure. For example, at
@@ -500,8 +511,8 @@ was satisfied on a different node, the o
 Slider should query the RM (or topology scripts?) to determine if nodes are still
 parts of the YARN cluster. </li>
 </ul>
-<h2 id="data-structures">Data Structures</h2>
-<h3 id="rolehistory">RoleHistory</h3>
+<h2 id="data-structures">Data Structures<a class="headerlink" href="#data-structures" title="Permanent link">&para;</a></h2>
+<h3 id="rolehistory">RoleHistory<a class="headerlink" href="#rolehistory" title="Permanent link">&para;</a></h3>
 <div class="codehilite"><pre><span class="n">startTime</span><span class="o">:</span> <span class="n">long</span>
 <span class="n">saveTime</span><span class="o">:</span> <span class="n">long</span>
 <span class="n">dirty</span><span class="o">:</span> <span class="n">boolean</span>
@@ -513,7 +524,7 @@ parts of the YARN cluster. </li>
 
 
 <p>This is the aggregate data structure that is persisted to/from file</p>
-<h3 id="nodemap">NodeMap</h3>
+<h3 id="nodemap">NodeMap<a class="headerlink" href="#nodemap" title="Permanent link">&para;</a></h3>
 <div class="codehilite"><pre><span class="n">clusterNodes</span><span class="o">:</span> <span class="n">Map</span><span class="o">:</span> <span class="n">NodeId</span> <span class="o">-&gt;</span> <span class="n">NodeInstance</span>
 <span class="n">clusterNodes</span><span class="o">():</span> <span class="n">Iterable</span><span class="o">&lt;</span><span class="n">NodeInstance</span><span class="o">&gt;</span>
 <span class="n">getOrCreate</span><span class="o">(</span><span class="n">NodeId</span><span class="o">):</span> <span class="n">NodeInstance</span>
@@ -521,7 +532,7 @@ parts of the YARN cluster. </li>
 
 
 <p>Maps a YARN NodeID record to a Slider <code>NodeInstance</code> structure</p>
-<h3 id="nodeinstance">NodeInstance</h3>
+<h3 id="nodeinstance">NodeInstance<a class="headerlink" href="#nodeinstance" title="Permanent link">&para;</a></h3>
 <p>Every node in the YARN cluster is modeled as an ragged array of <code>NodeEntry</code> instances, indexed
 by role index -</p>
 <div class="codehilite"><pre><span class="n">NodeEntry</span><span class="p">[</span><span class="n">roles</span><span class="p">]</span>
@@ -535,7 +546,7 @@ by role index -</p>
 
 <p>This could be implemented in a map or an indexed array; the array is more
 efficient but it does mandate that the number of roles are bounded and fixed.</p>
-<h3 id="nodeentry">NodeEntry</h3>
+<h3 id="nodeentry">NodeEntry<a class="headerlink" href="#nodeentry" title="Permanent link">&para;</a></h3>
 <p>Records the details about all of a roles containers on a node. The
 <code>active</code> field records the number of containers currently active.</p>
 <div class="codehilite"><pre><span class="n">active</span><span class="o">:</span> <span class="n">int</span>
@@ -556,9 +567,9 @@ responses (which may come from pre-resta
 unexpected container release responses as failures.</p>
 <p>The <code>active</code> counter is only decremented after a container release response
 has been received.</p>
-<h3 id="rolestatus">RoleStatus</h3>
+<h3 id="rolestatus">RoleStatus<a class="headerlink" href="#rolestatus" title="Permanent link">&para;</a></h3>
 <p>This is the existing <code>org.apache.hoya.yarn.appmaster.state.RoleStatus</code> class</p>
-<h3 id="rolelist">RoleList</h3>
+<h3 id="rolelist">RoleList<a class="headerlink" href="#rolelist" title="Permanent link">&para;</a></h3>
 <p>A list mapping role to int enum is needed to index NodeEntry elements in
 the NodeInstance arrays. Although such an enum is already implemented in the Slider
 Providers, explicitly serializing and deserializing it would make
@@ -566,7 +577,7 @@ the persistent structure easier to parse
 to changes in the number or position of roles.</p>
 <p>This list could also retain information about recently used/released nodes,
 so that the selection of containers to request could shortcut a search</p>
-<h3 id="containerpriority">ContainerPriority</h3>
+<h3 id="containerpriority">ContainerPriority<a class="headerlink" href="#containerpriority" title="Permanent link">&para;</a></h3>
 <p>The container priority field (a 32 bit integer) is used by Slider (0.5.x)
 to index the specific role in a container so as to determine which role
 has been offered in a container allocation message, and which role has
@@ -584,7 +595,7 @@ The main requirement  is: not have &gt;
 which places an upper bound on the size of the Application Instance.</p>
 <p>The splitting and merging will be implemented in a ContainerPriority class,
 for uniform access.</p>
-<h3 id="outstandingrequest">OutstandingRequest</h3>
+<h3 id="outstandingrequest">OutstandingRequest<a class="headerlink" href="#outstandingrequest" title="Permanent link">&para;</a></h3>
 <p>Tracks an outstanding request. This is used to correlate an allocation response
 (whose Container Priority file is used to locate this request), with the
 node and role used in the request.</p>
@@ -598,7 +609,7 @@ node and role used in the request.</p>
 
 <p>The node identifier may be null -which indicates that a request was made without
 a specific target node</p>
-<h3 id="outstandingrequesttracker">OutstandingRequestTracker</h3>
+<h3 id="outstandingrequesttracker">OutstandingRequestTracker<a class="headerlink" href="#outstandingrequesttracker" title="Permanent link">&para;</a></h3>
 <p>Contains a map from requestID to the specific <code>OutstandingRequest</code> made,
 and generates the request ID</p>
 <div class="codehilite"><pre><span class="n">nextRequestId</span><span class="o">:</span> <span class="n">int</span>
@@ -617,7 +628,7 @@ and generates the request ID</p>
 
 <p>The list operation can be implemented inefficiently unless it is found
 to be important -if so a more complex structure will be needed.</p>
-<h3 id="availablenodes">AvailableNodes</h3>
+<h3 id="availablenodes">AvailableNodes<a class="headerlink" href="#availablenodes" title="Permanent link">&para;</a></h3>
 <p>This is a field in <code>RoleHistory</code></p>
 <div class="codehilite"><pre><span class="n">availableNodes</span><span class="o">:</span> <span class="n">List</span><span class="o">&lt;</span><span class="n">NodeInstance</span><span class="o">&gt;[]</span>
 </pre></div>
@@ -654,7 +665,7 @@ as the target for the next resource requ
 <p>Strategy #1 is simpler; Strategy #2 <em>may</em> decrease the affinity of component placement,
 as the AM will be explicitly requesting an instance on a node which it knows
 is not running an instance of that role.</p>
-<h4 id="issue-what-to-do-about-failing-nodes">ISSUE What to do about failing nodes?</h4>
+<h4 id="issue-what-to-do-about-failing-nodes">ISSUE What to do about failing nodes?<a class="headerlink" href="#issue-what-to-do-about-failing-nodes" title="Permanent link">&para;</a></h4>
 <p>Should a node whose container just failed be placed at the
 top of the stack, ready for the next request? </p>
 <p>If the container failed due to an unexpected crash in the application, asking
@@ -664,10 +675,10 @@ back a new role instance on that machine
 will not be satisfied by that node.</p>
 <p>If there is a problem with the node, such that containers repeatedly fail on it,
 then re-requesting containers on it will amplify the damage.</p>
-<h2 id="actions">Actions</h2>
-<h3 id="bootstrap">Bootstrap</h3>
+<h2 id="actions">Actions<a class="headerlink" href="#actions" title="Permanent link">&para;</a></h2>
+<h3 id="bootstrap">Bootstrap<a class="headerlink" href="#bootstrap" title="Permanent link">&para;</a></h3>
 <p>Persistent Role History file not found; empty data structures created.</p>
-<h3 id="restart">Restart</h3>
+<h3 id="restart">Restart<a class="headerlink" href="#restart" title="Permanent link">&para;</a></h3>
 <p>When starting an Application Instance, the Role History should be loaded. </p>
 <p>If the history is missing <em>or cannot be loaded for any reason</em>,
 Slider must revert to the bootstrap actions.</p>
@@ -710,7 +721,7 @@ The state of the Application Instance af
 
 <p>After this operation, the structures are purged with all out of date entries,
 and the available node list contains a sorted list of the remainder.</p>
-<h3 id="am-restart">AM Restart</h3>
+<h3 id="am-restart">AM Restart<a class="headerlink" href="#am-restart" title="Permanent link">&para;</a></h3>
 <p>1: Create the initial data structures as a normal start operation
 2: update the structure with the list of live nodes, removing those nodes
 from the list of available nodes</p>
@@ -730,7 +741,7 @@ from the list of available nodes</p>
 
 <p>There's no need to resort the available node list —all that has happened
 is that some entries have been removed</p>
-<h3 id="teardown">Teardown</h3>
+<h3 id="teardown">Teardown<a class="headerlink" href="#teardown" title="Permanent link">&para;</a></h3>
 <ol>
 <li>If dirty, save role history to its file.</li>
 <li>Issue release requests</li>
@@ -743,7 +754,7 @@ the RoleHistory data to have been writte
 should assume that the history was saved to disk at some point during the life
 of the Application Instance —ideally after the most recent change, and that the information
 in it is only an approximate about what the previous state of the Application Instance was.</p>
-<h3 id="flex-requesting-a-container-in-role-role">Flex: Requesting a container in role <code>role</code></h3>
+<h3 id="flex-requesting-a-container-in-role-role">Flex: Requesting a container in role <code>role</code><a class="headerlink" href="#flex-requesting-a-container-in-role-role" title="Permanent link">&para;</a></h3>
 <div class="codehilite"><pre><span class="n">node</span> <span class="p">=</span> <span class="n">availableNodes</span><span class="p">[</span><span class="n">roleId</span><span class="p">].</span><span class="n">pop</span><span class="p">()</span> 
 <span class="k">if</span> <span class="n">node</span> !<span class="p">=</span> <span class="n">null</span> <span class="p">:</span>
   <span class="n">node</span><span class="p">.</span><span class="n">nodeEntry</span><span class="p">[</span><span class="n">roleId</span><span class="p">].</span><span class="n">requested</span><span class="o">++</span><span class="p">;</span>
@@ -771,7 +782,7 @@ As a result, the request will be downgra
 request —an acceptable degradation on an Application Instance where all the other entries
 in the nodemap have instances of that specific node —but not when there are
 empty nodes. </p>
-<h4 id="solutions">Solutions</h4>
+<h4 id="solutions">Solutions<a class="headerlink" href="#solutions" title="Permanent link">&para;</a></h4>
 <ol>
 <li>
 <p>Add some randomness in the search of the datastructure, rather than simply
@@ -790,7 +801,7 @@ even though it was recently requested".<
 isn't currently satisfying requests.</p>
 </li>
 </ol>
-<h4 id="history-issues">History Issues</h4>
+<h4 id="history-issues">History Issues<a class="headerlink" href="#history-issues" title="Permanent link">&para;</a></h4>
 <p>Without using that history, there is a risk that a very old assignment
 is used in place of a recent one and the value of locality decreased.</p>
 <p>But there are consequences:</p>
@@ -808,7 +819,7 @@ be rebuilt at startup time.</p>
 <p>The proposed <code>recentlyReleasedList</code> addresses this, though it creates
 another data structure to maintain and rebuild at startup time
 from the last-used fields in the node entries.</p>
-<h3 id="am-callback-oncontainersallocated">AM Callback : onContainersAllocated</h3>
+<h3 id="am-callback-oncontainersallocated">AM Callback : onContainersAllocated<a class="headerlink" href="#am-callback-oncontainersallocated" title="Permanent link">&para;</a></h3>
 <div class="codehilite"><pre><span class="n">void</span> <span class="n">onContainersAllocated</span><span class="p">(</span><span class="n">List</span><span class="o">&lt;</span><span class="n">Container</span><span class="o">&gt;</span> <span class="n">allocatedContainers</span><span class="p">)</span>
 </pre></div>
 
@@ -881,7 +892,7 @@ re-inserted into the AvailableNodes list
 in the total ordering of the list.</p>
 </li>
 </ol>
-<h3 id="nmclientasync-callback-oncontainerstarted">NMClientAsync Callback:  onContainerStarted()</h3>
+<h3 id="nmclientasync-callback-oncontainerstarted">NMClientAsync Callback:  onContainerStarted()<a class="headerlink" href="#nmclientasync-callback-oncontainerstarted" title="Permanent link">&para;</a></h3>
 <div class="codehilite"><pre><span class="n">onContainerStarted</span><span class="p">(</span><span class="n">ContainerId</span> <span class="n">containerId</span><span class="p">)</span>
 </pre></div>
 
@@ -890,12 +901,12 @@ in the total ordering of the list.</p>
 of starting containers, moving it into the map of live nodes; the counters
 in the associated <code>RoleInstance</code> are updated accordingly; the node entry
 adjusted to indicate it has one more live node and one less starting node.</p>
-<h3 id="nmclientasync-callback-oncontainerstartfailed">NMClientAsync Callback:  onContainerStartFailed()</h3>
+<h3 id="nmclientasync-callback-oncontainerstartfailed">NMClientAsync Callback:  onContainerStartFailed()<a class="headerlink" href="#nmclientasync-callback-oncontainerstartfailed" title="Permanent link">&para;</a></h3>
 <p>The AM uses this as a signal to remove the container from the list
 of starting containers —the count of starting containers for the relevant
 NodeEntry is decremented. If the node is now available for instances of this
 container, it is returned to the queue of available nodes.</p>
-<h3 id="flex-releasing-a-component-instance-from-the-application-instance">Flex: Releasing a Component instance from the Application Instance</h3>
+<h3 id="flex-releasing-a-component-instance-from-the-application-instance">Flex: Releasing a Component instance from the Application Instance<a class="headerlink" href="#flex-releasing-a-component-instance-from-the-application-instance" title="Permanent link">&para;</a></h3>
 <p>Simple strategy: find a node with at least one active container of that type</p>
 <div class="codehilite"><pre><span class="n">select</span> <span class="n">a</span> <span class="n">node</span> <span class="n">N</span> <span class="n">in</span> <span class="n">nodemap</span> <span class="n">where</span> <span class="k">for</span> <span class="n">NodeEntry</span><span class="p">[</span><span class="n">roleId</span><span class="p">]:</span> <span class="n">active</span> <span class="o">&gt;</span> <span class="n">releasing</span><span class="p">;</span> 
 <span class="n">nodeentry</span> <span class="p">=</span> <span class="n">node</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">roleId</span><span class="p">)</span>
@@ -919,7 +930,7 @@ must: be of the target role, and not alr
 <li>The (existing) <code>containersBeingReleased</code> Map has the container inserted into it</li>
 </ol>
 <p>After the AM processes the request, it triggers a callback</p>
-<h3 id="am-callback-oncontainerscompleted">AM callback onContainersCompleted:</h3>
+<h3 id="am-callback-oncontainerscompleted">AM callback onContainersCompleted:<a class="headerlink" href="#am-callback-oncontainerscompleted" title="Permanent link">&para;</a></h3>
 <div class="codehilite"><pre><span class="n">void</span> <span class="n">onContainersCompleted</span><span class="p">(</span><span class="n">List</span><span class="o">&lt;</span><span class="n">ContainerStatus</span><span class="o">&gt;</span> <span class="n">completedContainers</span><span class="p">)</span>
 </pre></div>
 
@@ -972,7 +983,7 @@ on that node, it will have been inserted
 As a result, it is highly likely that a new container will be requested on 
 the same node. (The only way a node the list would be newer is 
 be if other containers were completed in the same callback)</p>
-<h3 id="implementation-notes">Implementation Notes</h3>
+<h3 id="implementation-notes">Implementation Notes<a class="headerlink" href="#implementation-notes" title="Permanent link">&para;</a></h3>
 <p>Notes made while implementing the design.</p>
 <p><code>OutstandingRequestTracker</code> should also track requests made with
 no target node; this makes seeing what is going on easier. <code>ARMClientImpl</code>
@@ -999,13 +1010,13 @@ have been released?  I.e. Before YARN ha
 <p>RoleStats were removed —left in app state. Although the rolestats would
 belong here, leaving them where they were reduced the amount of change
 in the <code>AppState</code> class, so risk of something breaking.</p>
-<h2 id="miniyarncluster-node-ids">MiniYARNCluster node IDs</h2>
+<h2 id="miniyarncluster-node-ids">MiniYARNCluster node IDs<a class="headerlink" href="#miniyarncluster-node-ids" title="Permanent link">&para;</a></h2>
 <p>Mini YARN cluster NodeIDs all share the same hostname , at least when running
 against file://; so mini tests with &gt;1 NM don't have a 1:1 mapping of
 <code>NodeId:NodeInstance</code>. What will happen is that 
 <code>NodeInstance getOrCreateNodeInstance(Container container) '
 will always return the same (now shared)</code>NodeInstance`.</p>
-<h2 id="releasing-containers-when-shrinking-an-application-instance">Releasing Containers when shrinking an Application Instance</h2>
+<h2 id="releasing-containers-when-shrinking-an-application-instance">Releasing Containers when shrinking an Application Instance<a class="headerlink" href="#releasing-containers-when-shrinking-an-application-instance" title="Permanent link">&para;</a></h2>
 <p>When identifying instances to release in a bulk downscale operation, the full
 list of targets must be identified together. This is not just to eliminate
 multiple scans of the data structures, but because the containers are not
@@ -1023,12 +1034,12 @@ a new datastructure is created to identi
 (e.g a list of ContainerIds under each NodeEntry), or we add an index reference
 in a RoleInstance that identifies the node. We already effectively have that
 in the container</p>
-<h3 id="dropping-any-available-nodes-that-are-busy">dropping any available nodes that are busy</h3>
+<h3 id="dropping-any-available-nodes-that-are-busy">dropping any available nodes that are busy<a class="headerlink" href="#dropping-any-available-nodes-that-are-busy" title="Permanent link">&para;</a></h3>
 <p>When scanning the available list, any nodes that are no longer idle for that
 role should be dropped from the list.</p>
 <p>This can happen when an instance was allocated on a different node from
 that requested.</p>
-<h3 id="finding-a-node-when-a-component-has-some-live-instances-and-no-available-hosts-known-to">Finding a node when a component has some live instances and no available hosts known to</h3>
+<h3 id="finding-a-node-when-a-component-has-some-live-instances-and-no-available-hosts-known-to">Finding a node when a component has some live instances and no available hosts known to<a class="headerlink" href="#finding-a-node-when-a-component-has-some-live-instances-and-no-available-hosts-known-to" title="Permanent link">&para;</a></h3>
 <p>Slider.</p>
 <p>One condition found during testing is the following: </p>
 <ol>
@@ -1054,7 +1065,7 @@ which are not running an instance of the
 the request would have to omit those nodes for which an allocation has already been
 made off the available list and yet for which no container has yet been
 granted]. </p>
-<h2 id="reworked-outstanding-request-tracker">Reworked Outstanding Request Tracker</h2>
+<h2 id="reworked-outstanding-request-tracker">Reworked Outstanding Request Tracker<a class="headerlink" href="#reworked-outstanding-request-tracker" title="Permanent link">&para;</a></h2>
 <p>The reworked request tracker behaves as follows</p>
 <ol>
 <li>outstanding requests with specific placements are tracked by <code>(role, hostname)</code></li>
@@ -1068,7 +1079,7 @@ then sorted.</li>
 </ol>
 <p>This strategy returns unused hosts to the list of possible hosts, while retaining
 the ordering of that list in most-recent-first.</p>
-<h3 id="weaknesses_1">Weaknesses</h3>
+<h3 id="weaknesses_1">Weaknesses<a class="headerlink" href="#weaknesses_1" title="Permanent link">&para;</a></h3>
 <p>if one or more container requests cannot be satisifed, then all the hosts in
 the set of outstanding requests will be retained, so all these hosts in the
 will be considered unavailable for new location-specific requests.
@@ -1080,7 +1091,7 @@ same priority (i.e. same Slider Role ID)
 of instances of the target role were decreated during a flex such that
 the placement could now be satisfied on the target host. This is not considered
 a significant problem.</p>
-<h1 id="persistence_1">Persistence</h1>
+<h1 id="persistence_1">Persistence<a class="headerlink" href="#persistence_1" title="Permanent link">&para;</a></h1>
 <p>The initial implementation uses the JSON-formatted Avro format; while significantly
 less efficient than a binary format, it is human-readable</p>
 <p>Here are sequence of entries from a test run on a single node YARN cluster; running 1 HBase Master
@@ -1134,7 +1145,7 @@ No region servers are yet live.</p>
 
 <p>The <code>last_used</code> timestamps will not be changed until the Application Instance is shrunk or restarted, as the <code>active</code> flag being set
 implies that the server is running both roles at the save time of <code>1384183512217</code>.</p>
-<h2 id="resolved-issues">Resolved issues</h2>
+<h2 id="resolved-issues">Resolved issues<a class="headerlink" href="#resolved-issues" title="Permanent link">&para;</a></h2>
 <blockquote>
 <p>How best to distinguish at startup time from nodes used recently
 from nodes used some period before? Should the RoleHistory simply forget

Modified: websites/staging/slider/trunk/content/design/specification/cli-actions.html
==============================================================================
--- websites/staging/slider/trunk/content/design/specification/cli-actions.html (original)
+++ websites/staging/slider/trunk/content/design/specification/cli-actions.html Mon Nov  2 17:24:47 2015
@@ -155,7 +155,7 @@
   <div style="text-align: center">
     <h1><a href="/index.html">Apache Slider (incubating)</a></h1>
     <hr>
-Latest release: <strong>0.80.0-incubating</strong><br>
+Latest release: <strong>0.81.1-incubating</strong><br>
     <br>
     <a id="download-button-sidebar" class="btn btn-success btn-block" href="/downloads/" role="button">Download</a>
   </div>
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
 
     <h1 class="title"></h1>
 
-    <!---
+    <style type="text/css">
+/* The following code is added by mdx_elementid.py
+   It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+  visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink, h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink, dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
    Licensed to the Apache Software Foundation (ASF) under one or more
    contributor license agreements.  See the NOTICE file distributed with
    this work for additional information regarding copyright ownership.
@@ -185,29 +196,29 @@ Latest release: <strong>0.80.0-incubatin
    limitations under the License.
 -->
 
-<h1 id="apache-slider-cli-actions">Apache Slider CLI Actions</h1>
-<h2 id="important">Important</h2>
+<h1 id="apache-slider-cli-actions">Apache Slider CLI Actions<a class="headerlink" href="#apache-slider-cli-actions" title="Permanent link">&para;</a></h1>
+<h2 id="important">Important<a class="headerlink" href="#important" title="Permanent link">&para;</a></h2>
 <ol>
 <li>This document is still being updated from the original hoya design</li>
 <li>The new cluster model of separated specification files for internal, resource and application configuration
 has not been incorporated.</li>
 <li>What is up to date is the CLI command list and arguments</li>
 </ol>
-<h2 id="client-configuration">client configuration</h2>
+<h2 id="client-configuration">client configuration<a class="headerlink" href="#client-configuration" title="Permanent link">&para;</a></h2>
 <p>As well as the CLI options, the <code>conf/slider-client.xml</code> XML file can define arguments used to communicate with the Application instance</p>
-<h4 id="fsdefaultfs"><code>fs.defaultFS</code></h4>
+<h4 id="fsdefaultfs"><code>fs.defaultFS</code><a class="headerlink" href="#fsdefaultfs" title="Permanent link">&para;</a></h4>
 <p>Equivalent to setting the filesystem with <code>--filesystem</code></p>
-<h2 id="common">Common</h2>
-<h3 id="system-properties">System Properties</h3>
+<h2 id="common">Common<a class="headerlink" href="#common" title="Permanent link">&para;</a></h2>
+<h3 id="system-properties">System Properties<a class="headerlink" href="#system-properties" title="Permanent link">&para;</a></h3>
 <p>Arguments of the form <code>-S key=value</code> define JVM system properties.</p>
 <p>These are supported primarily to define options needed for some Kerberos configurations.</p>
-<h3 id="definitions">Definitions</h3>
+<h3 id="definitions">Definitions<a class="headerlink" href="#definitions" title="Permanent link">&para;</a></h3>
 <p>Arguments of the form <code>-D key=value</code> define JVM system properties.</p>
 <p>These can define client options that are not set in <code>conf/slider-client.xml</code> - or to override them.</p>
-<h3 id="cluster-names">Cluster names</h3>
+<h3 id="cluster-names">Cluster names<a class="headerlink" href="#cluster-names" title="Permanent link">&para;</a></h3>
 <p>All actions that must take an instance name will fail with <code>EXIT_UNKNOWN_INSTANCE</code>
 if one is not provided.</p>
-<h2 id="action-build">Action: Build</h2>
+<h2 id="action-build">Action: Build<a class="headerlink" href="#action-build" title="Permanent link">&para;</a></h2>
 <p>Builds a cluster -creates all the on-filesystem datastructures, and generates a cluster description
 that is both well-defined and deployable -<em>but does not actually start the cluster</em></p>
 <div class="codehilite"><pre><span class="n">build</span> <span class="p">(</span><span class="n">instancename</span><span class="p">,</span>
@@ -226,7 +237,7 @@ that is both well-defined and deployable
 </pre></div>
 
 
-<h4 id="preconditions">Preconditions</h4>
+<h4 id="preconditions">Preconditions<a class="headerlink" href="#preconditions" title="Permanent link">&para;</a></h4>
 <p>(Note that the ordering of these preconditions is not guaranteed to remain constant)</p>
 <p>The instance name is valid</p>
 <div class="codehilite"><pre><span class="k">if</span> <span class="n">not</span> <span class="n">valid</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">name</span><span class="p">(</span><span class="n">instancename</span><span class="p">)</span> <span class="p">:</span> <span class="n">raise</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_COMMAND_ARGUMENT_ERROR</span><span class="p">)</span>
@@ -260,7 +271,7 @@ If it exists when another process wishes
 afterwards. A process attempting to acquire the writelock must check for the existence of this file before AND after creating the
 writelock file, failing if its present. This retains a small race condition: a second or later reader may still be reading the data
 when a process successfully acquires the write lock. If this proves to be an issue, a stricter model could be implemented, with each reading process creating a unique named readlock- file.</p>
-<h4 id="postconditions">Postconditions</h4>
+<h4 id="postconditions">Postconditions<a class="headerlink" href="#postconditions" title="Permanent link">&para;</a></h4>
 <p>All the instance directories exist</p>
 <div class="codehilite"><pre><span class="n">is</span><span class="o">-</span><span class="n">dir</span><span class="p">(</span><span class="n">HDFS</span><span class="o">&#39;</span><span class="p">,</span> <span class="n">instance</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">HDFS</span><span class="o">&#39;</span><span class="p">,</span> <span class="n">instancename</span><span class="p">))</span>
 <span class="n">is</span><span class="o">-</span><span class="n">dir</span><span class="p">(</span><span class="n">HDFS</span><span class="o">&#39;</span><span class="p">,</span> <span class="n">original</span><span class="o">-</span><span class="n">conf</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">HDFS</span><span class="o">&#39;</span><span class="p">,</span> <span class="n">instancename</span><span class="p">))</span>
@@ -354,14 +365,14 @@ the command-line supplied option will ov
 </pre></div>
 
 
-<h2 id="action-start">Action: Start</h2>
+<h2 id="action-start">Action: Start<a class="headerlink" href="#action-start" title="Permanent link">&para;</a></h2>
 <div class="codehilite"><pre><span class="n">start</span> <span class="o">&lt;</span><span class="n">instancename</span><span class="o">&gt;</span> <span class="p">[</span><span class="o">--</span><span class="n">wait</span> <span class="o">&lt;</span><span class="n">timeout</span><span class="o">&gt;</span><span class="p">]</span>
 </pre></div>
 
 
 <p>Takes an application instance with configuration and (possibly) data on disk, and
 attempts to create a live application with the specified number of nodes</p>
-<h4 id="preconditions_1">Preconditions</h4>
+<h4 id="preconditions_1">Preconditions<a class="headerlink" href="#preconditions_1" title="Permanent link">&para;</a></h4>
 <div class="codehilite"><pre><span class="k">if</span> <span class="n">not</span> <span class="n">valid</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">name</span><span class="p">(</span><span class="n">instancename</span><span class="p">)</span> <span class="p">:</span> <span class="n">raise</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_COMMAND_ARGUMENT_ERROR</span><span class="p">)</span>
 </pre></div>
 
@@ -383,7 +394,7 @@ attempts to create a live application wi
 </pre></div>
 
 
-<h3 id="postconditions_1">Postconditions</h3>
+<h3 id="postconditions_1">Postconditions<a class="headerlink" href="#postconditions_1" title="Permanent link">&para;</a></h3>
 <p>After the start operation has been performed, there is now a queued request in YARN
 for the chosen (how?) queue</p>
 <div class="codehilite"><pre><span class="n">YARN</span><span class="o">&#39;</span><span class="p">.</span><span class="n">Queues</span><span class="o">&#39;</span><span class="p">[</span><span class="n">amqueue</span><span class="p">]</span> <span class="p">=</span> <span class="n">YARN</span><span class="p">.</span><span class="n">Queues</span><span class="p">[</span><span class="n">amqueue</span><span class="p">]</span> <span class="o">+</span> <span class="p">[</span><span class="n">launch</span><span class="p">(</span>&quot;<span class="n">slider</span>&quot;<span class="p">,</span> <span class="n">instancename</span><span class="p">,</span> <span class="n">requirements</span><span class="p">,</span> <span class="n">context</span><span class="p">)]</span>
@@ -398,17 +409,17 @@ the application has failed</p>
 </pre></div>
 
 
-<h2 id="outcome-am-launched-state">Outcome: AM-launched state</h2>
+<h2 id="outcome-am-launched-state">Outcome: AM-launched state<a class="headerlink" href="#outcome-am-launched-state" title="Permanent link">&para;</a></h2>
 <p>Some time after the AM was queued, if the relevant
 prerequisites of the launch request are met, the AM will be deployed</p>
-<h4 id="preconditions_2">Preconditions</h4>
+<h4 id="preconditions_2">Preconditions<a class="headerlink" href="#preconditions_2" title="Permanent link">&para;</a></h4>
 <ul>
 <li>The resources referenced in HDFS (still) are accessible by the user</li>
 <li>The requested YARN memory and core requirements could be met on the YARN cluster and 
 specific YARN application queue.</li>
 <li>There is sufficient capacity in the YARN cluster to create a container for the AM.</li>
 </ul>
-<h4 id="postconditions_2">Postconditions</h4>
+<h4 id="postconditions_2">Postconditions<a class="headerlink" href="#postconditions_2" title="Permanent link">&para;</a></h4>
 <p>Define a YARN state at a specific time <code>t</code> as <code>YARN(t)</code>; the fact that
 an AM is launched afterwards</p>
 <p>The AM is deployed if there is some time <code>t</code> after the submission time <code>t0</code>
@@ -429,7 +440,7 @@ requests using resources as they become
 <p>For tests on a dedicated YARN cluster, a few tens of seconds appear to be enough
 for the AM-launched state to be reached, a failure to occur, or to conclude
 that the resource requirements are unsatisfiable.</p>
-<h2 id="outcome-am-started-state">Outcome: AM-started state</h2>
+<h2 id="outcome-am-started-state">Outcome: AM-started state<a class="headerlink" href="#outcome-am-started-state" title="Permanent link">&para;</a></h2>
 <p>A (usually short) time after the AM is launched, it should start</p>
 <ul>
 <li>The node hosting the container is working reliably</li>
@@ -444,13 +455,13 @@ with an error code.</li>
 </ul>
 <p>Node failures/command line failures are treated by YARN as an AM failure which
 will trigger a restart attempt -this may be on the same or a different node.</p>
-<h4 id="preconditions_3">preconditions</h4>
+<h4 id="preconditions_3">preconditions<a class="headerlink" href="#preconditions_3" title="Permanent link">&para;</a></h4>
 <p>The AM was launched at an earlier time, <code>t1</code></p>
 <div class="codehilite"><pre><span class="n">exists</span> <span class="n">t1</span> <span class="n">where</span> <span class="n">t1</span> <span class="o">&gt;</span> <span class="n">t0</span> <span class="n">and</span> <span class="n">am</span><span class="o">-</span><span class="n">launched</span><span class="p">(</span><span class="n">YARN</span><span class="p">(</span><span class="n">t1</span><span class="p">)</span>
 </pre></div>
 
 
-<h4 id="postconditions_3">Postconditions</h4>
+<h4 id="postconditions_3">Postconditions<a class="headerlink" href="#postconditions_3" title="Permanent link">&para;</a></h4>
 <p>The application is actually started if it is listed in the YARN application list
 as being in the state <code>RUNNING</code>, an RPC port has been registered with YARN (visible as the <code>rpcPort</code>
 attribute in the YARN Application Report,and that port is servicing RPC requests
@@ -466,7 +477,7 @@ from authenticated callers.</p>
 <p>A test for accepting cluster requests is querying the cluster status
 with <code>SliderClusterProtocol.getJSONClusterStatus()</code>. If this returns
 a parseable cluster description, the AM considers itself live.</p>
-<h2 id="outcome-applicaton-instance-operational-state">Outcome: Applicaton Instance operational state</h2>
+<h2 id="outcome-applicaton-instance-operational-state">Outcome: Applicaton Instance operational state<a class="headerlink" href="#outcome-applicaton-instance-operational-state" title="Permanent link">&para;</a></h2>
 <p>Once started, Slider enters the operational state of trying to keep the numbers
 of live role instances matching the numbers specified in the cluster specification.</p>
 <p>The AM must request the a container for each desired instance of a specific roles of the
@@ -476,7 +487,7 @@ the specific application roles on the al
 cluster size is dynamically updated.</p>
 <p>The AM releases containers when the cluster size is shrunk during a flex operation,
 or during teardown.</p>
-<h3 id="steady-state-condition">steady state condition</h3>
+<h3 id="steady-state-condition">steady state condition<a class="headerlink" href="#steady-state-condition" title="Permanent link">&para;</a></h3>
 <p>The steady state of a Slider cluster is that the number of live instances of a role,
 plus the number of requested instances , minus the number of instances for
 which release requests have been made must match that of the desired number.</p>
@@ -499,10 +510,10 @@ history of notifications received from Y
 <p>Slider does not consider it an error if the number of actual instances remains below
 the desired value (i.e. outstanding requests are not being satisfied) -this is
 an operational state of the cluster that Slider cannot address.</p>
-<h3 id="cluster-startup">Cluster startup</h3>
+<h3 id="cluster-startup">Cluster startup<a class="headerlink" href="#cluster-startup" title="Permanent link">&para;</a></h3>
 <p>On a healthy dedicated test cluster, the time for the requests to be satisfied is
 a few tens of seconds at most: a failure to achieve this state is a sign of a problem.</p>
-<h3 id="node-or-process-failure">Node or process failure</h3>
+<h3 id="node-or-process-failure">Node or process failure<a class="headerlink" href="#node-or-process-failure" title="Permanent link">&para;</a></h3>
 <p>After a container or node failure, a new container for a new instance of that role
 is requested.</p>
 <p>The failure count is incremented -it can be accessed via the <code>"role.failed.instances"</code>
@@ -518,7 +529,7 @@ option: <code>"slider.container.failure.
 </pre></div>
 
 
-<h3 id="instance-startup-failure">Instance startup failure</h3>
+<h3 id="instance-startup-failure">Instance startup failure<a class="headerlink" href="#instance-startup-failure" title="Permanent link">&para;</a></h3>
 <p>Startup failures are measured alongside general node failures.</p>
 <p>A container is deemed to have failed to start if either of the following conditions
 were met:</p>
@@ -537,10 +548,10 @@ startup failures differently from ongoin
 treated as a sign that the container is failing to launch the program reliably -
 either the generated command line is invalid, or the application is failing
 to run/exiting on or nearly immediately.</p>
-<h2 id="action-create">Action: Create</h2>
+<h2 id="action-create">Action: Create<a class="headerlink" href="#action-create" title="Permanent link">&para;</a></h2>
 <p>Create is simply <code>build</code> + <code>start</code> in sequence  - the postconditions from the first
 action are intended to match the preconditions of the second.</p>
-<h2 id="action-stop">Action: Stop</h2>
+<h2 id="action-stop">Action: Stop<a class="headerlink" href="#action-stop" title="Permanent link">&para;</a></h2>
 <div class="codehilite"><pre><span class="n">stop</span> <span class="n">instancename</span> <span class="p">[</span><span class="o">--</span><span class="n">wait</span> <span class="n">time</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">message</span> <span class="n">message</span><span class="p">]</span>
 </pre></div>
 
@@ -549,7 +560,7 @@ action are intended to match the precond
 cluster are stopped, leaving all the persistent state.</p>
 <p>The operation is intended to be idempotent: it is not an error if 
 <code>stop</code> is invoked on an already stopped cluster</p>
-<h4 id="preconditions_4">Preconditions</h4>
+<h4 id="preconditions_4">Preconditions<a class="headerlink" href="#preconditions_4" title="Permanent link">&para;</a></h4>
 <p>The cluster name is valid and it matches a known cluster </p>
 <div class="codehilite"><pre><span class="k">if</span> <span class="n">not</span> <span class="n">valid</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">name</span><span class="p">(</span><span class="n">instancename</span><span class="p">)</span> <span class="p">:</span> <span class="n">raise</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_COMMAND_ARGUMENT_ERROR</span><span class="p">)</span>
 
@@ -558,7 +569,7 @@ cluster are stopped, leaving all the per
 </pre></div>
 
 
-<h4 id="postconditions_4">Postconditions</h4>
+<h4 id="postconditions_4">Postconditions<a class="headerlink" href="#postconditions_4" title="Permanent link">&para;</a></h4>
 <p>If the cluster was running, an RPC call has been sent to it <code>stopCluster(message)</code></p>
 <p>If the <code>--wait</code> argument specified a wait time, then the command will block
 until the cluster has finished or the wait time was exceeded. </p>
@@ -569,7 +580,7 @@ YARN logs as the reason the cluster was
 </pre></div>
 
 
-<h2 id="action-flex">Action: Flex</h2>
+<h2 id="action-flex">Action: Flex<a class="headerlink" href="#action-flex" title="Permanent link">&para;</a></h2>
 <p>Flex the cluster size: add or remove roles. </p>
 <div class="codehilite"><pre><span class="n">flex</span> <span class="n">instancename</span> 
 <span class="n">components</span><span class="p">:</span><span class="n">List</span><span class="p">[(</span><span class="n">String</span><span class="p">,</span> <span class="n">int</span><span class="p">)]</span>
@@ -581,13 +592,13 @@ YARN logs as the reason the cluster was
 <li>if the cluster is running, it is given the new cluster specification,
 which will change the desired steady-state of the application</li>
 </ol>
-<h4 id="preconditions_5">Preconditions</h4>
+<h4 id="preconditions_5">Preconditions<a class="headerlink" href="#preconditions_5" title="Permanent link">&para;</a></h4>
 <div class="codehilite"><pre><span class="k">if</span> <span class="n">not</span> <span class="n">is</span><span class="o">-</span><span class="n">file</span><span class="p">(</span><span class="n">HDFS</span><span class="p">,</span> <span class="n">cluster</span><span class="o">-</span><span class="n">json</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">HDFS</span><span class="p">,</span> <span class="n">instancename</span><span class="p">))</span> <span class="p">:</span>
     <span class="n">raise</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_UNKNOWN_INSTANCE</span><span class="p">)</span>
 </pre></div>
 
 
-<h4 id="postconditions_5">Postconditions</h4>
+<h4 id="postconditions_5">Postconditions<a class="headerlink" href="#postconditions_5" title="Permanent link">&para;</a></h4>
 <div class="codehilite"><pre><span class="n">let</span> <span class="n">originalSpec</span> <span class="p">=</span> <span class="n">data</span><span class="p">(</span><span class="n">HDFS</span><span class="p">,</span> <span class="n">cluster</span><span class="o">-</span><span class="n">json</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">HDFS</span><span class="p">,</span> <span class="n">instancename</span><span class="p">))</span>
 
 <span class="n">let</span> <span class="n">updatedSpec</span> <span class="p">=</span> <span class="n">originalspec</span> <span class="n">where</span><span class="p">:</span>
@@ -599,7 +610,7 @@ which will change the desired steady-sta
 </pre></div>
 
 
-<h4 id="am-actions-on-flex">AM actions on flex</h4>
+<h4 id="am-actions-on-flex">AM actions on flex<a class="headerlink" href="#am-actions-on-flex" title="Permanent link">&para;</a></h4>
 <div class="codehilite"><pre><span class="n">boolean</span> <span class="n">SliderAppMaster</span><span class="p">.</span><span class="n">flexCluster</span><span class="p">(</span><span class="n">ClusterDescription</span> <span class="n">updatedSpec</span><span class="p">)</span>
 </pre></div>
 
@@ -608,12 +619,12 @@ which will change the desired steady-sta
 then <code>AppState</code> is updated with the new desired role counts. The operation will
 return once all requests to add or remove role instances have been queued,
 and be <code>True</code> iff the desired steady state of the cluster has been changed.</p>
-<h4 id="preconditions_6">Preconditions</h4>
+<h4 id="preconditions_6">Preconditions<a class="headerlink" href="#preconditions_6" title="Permanent link">&para;</a></h4>
 <div class="codehilite"><pre>  <span class="n">well</span><span class="o">-</span><span class="n">defined</span><span class="o">-</span><span class="n">application</span><span class="o">-</span><span class="n">instance</span><span class="p">(</span><span class="n">HDFS</span><span class="p">,</span> <span class="n">updatedSpec</span><span class="p">)</span>
 </pre></div>
 
 
-<h4 id="postconditions_6">Postconditions</h4>
+<h4 id="postconditions_6">Postconditions<a class="headerlink" href="#postconditions_6" title="Permanent link">&para;</a></h4>
 <div class="codehilite"><pre><span class="n">forall</span> <span class="n">role</span> <span class="n">in</span> <span class="n">AppState</span><span class="p">.</span><span class="n">Roles</span><span class="p">.</span><span class="n">keys</span><span class="p">:</span>
     <span class="n">AppState</span><span class="o">&#39;</span><span class="p">.</span><span class="n">Roles</span><span class="o">&#39;</span><span class="p">[</span><span class="n">role</span><span class="p">].</span><span class="n">desiredCount</span> <span class="p">=</span> <span class="n">updatedSpec</span><span class="p">[</span><span class="n">roles</span><span class="p">][</span>&quot;<span class="n">yarn</span><span class="p">.</span><span class="n">component</span><span class="p">.</span><span class="n">instances</span>&quot;<span class="p">]</span>
 <span class="n">result</span> <span class="p">=</span> <span class="n">AppState</span><span class="o">&#39;</span> !<span class="p">=</span> <span class="n">AppState</span>
@@ -624,35 +635,35 @@ and be <code>True</code> iff the desired
 case the relevant requests will have been queued by the completion of the
 action. It is not possible to state whether or when the requests will be
 satisfied.</p>
-<h2 id="action-destroy">Action: Destroy</h2>
+<h2 id="action-destroy">Action: Destroy<a class="headerlink" href="#action-destroy" title="Permanent link">&para;</a></h2>
 <p>Idempotent operation to destroy a stopped cluster -it succeeds if the 
 cluster has already been destroyed/is unknown, but not if it is
 actually running.</p>
-<h4 id="preconditions_7">Preconditions</h4>
+<h4 id="preconditions_7">Preconditions<a class="headerlink" href="#preconditions_7" title="Permanent link">&para;</a></h4>
 <div class="codehilite"><pre><span class="k">if</span> <span class="n">not</span> <span class="n">valid</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">name</span><span class="p">(</span><span class="n">instancename</span><span class="p">)</span> <span class="p">:</span> <span class="n">raise</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_COMMAND_ARGUMENT_ERROR</span><span class="p">)</span>
 
 <span class="k">if</span> <span class="n">slider</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">live</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">instancename</span><span class="p">)</span> <span class="p">:</span> <span class="n">raise</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_CLUSTER_IN_USE</span><span class="p">)</span>
 </pre></div>
 
 
-<h4 id="postconditions_7">Postconditions</h4>
+<h4 id="postconditions_7">Postconditions<a class="headerlink" href="#postconditions_7" title="Permanent link">&para;</a></h4>
 <p>The cluster directory and all its children do not exist</p>
 <div class="codehilite"><pre><span class="n">not</span> <span class="n">is</span><span class="o">-</span><span class="n">dir</span><span class="p">(</span><span class="n">HDFS</span><span class="o">&#39;</span><span class="p">,</span> <span class="n">application</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">HDFS</span><span class="o">&#39;</span><span class="p">,</span> <span class="n">instancename</span><span class="p">))</span>
 </pre></div>
 
 
-<h2 id="action-status">Action: Status</h2>
+<h2 id="action-status">Action: Status<a class="headerlink" href="#action-status" title="Permanent link">&para;</a></h2>
 <div class="codehilite"><pre><span class="n">status</span> <span class="n">instancename</span> <span class="p">[</span><span class="o">--</span><span class="n">out</span> <span class="n">outfile</span><span class="p">]</span>
 2
 </pre></div>
 
 
-<h4 id="preconditions_8">Preconditions</h4>
+<h4 id="preconditions_8">Preconditions<a class="headerlink" href="#preconditions_8" title="Permanent link">&para;</a></h4>
 <div class="codehilite"><pre><span class="k">if</span> <span class="n">not</span> <span class="n">slider</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">live</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">instancename</span><span class="p">)</span> <span class="p">:</span> <span class="n">raise</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_UNKNOWN_INSTANCE</span><span class="p">)</span>
 </pre></div>
 
 
-<h4 id="postconditions_8">Postconditions</h4>
+<h4 id="postconditions_8">Postconditions<a class="headerlink" href="#postconditions_8" title="Permanent link">&para;</a></h4>
 <p>The status of the application has been successfully queried and printed out:</p>
 <div class="codehilite"><pre><span class="n">let</span> <span class="n">status</span> <span class="p">=</span> <span class="n">slider</span><span class="o">-</span><span class="n">live</span><span class="o">-</span><span class="n">instances</span><span class="p">(</span><span class="n">YARN</span><span class="p">).</span><span class="n">rpcPort</span><span class="p">.</span><span class="n">getJSONClusterStatus</span><span class="p">()</span>
 </pre></div>
@@ -669,18 +680,18 @@ actually running.</p>
 </pre></div>
 
 
-<h2 id="action-exists">Action: Exists</h2>
+<h2 id="action-exists">Action: Exists<a class="headerlink" href="#action-exists" title="Permanent link">&para;</a></h2>
 <p>This probes for a named cluster being defined or actually being in the running
 state.</p>
 <p>In the running state; it is essentially the status
 operation with only the exit code returned</p>
-<h4 id="preconditions_9">Preconditions</h4>
+<h4 id="preconditions_9">Preconditions<a class="headerlink" href="#preconditions_9" title="Permanent link">&para;</a></h4>
 <div class="codehilite"><pre><span class="k">if</span> <span class="n">not</span> <span class="n">is</span><span class="o">-</span><span class="n">file</span><span class="p">(</span><span class="n">HDFS</span><span class="p">,</span> <span class="n">application</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">HDFS</span><span class="p">,</span> <span class="n">instancename</span><span class="p">))</span> <span class="p">:</span>
     <span class="n">raise</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_UNKNOWN_INSTANCE</span><span class="p">)</span>
 </pre></div>
 
 
-<h4 id="postconditions_9">Postconditions</h4>
+<h4 id="postconditions_9">Postconditions<a class="headerlink" href="#postconditions_9" title="Permanent link">&para;</a></h4>
 <p>The operation succeeds if the cluster is running and the RPC call returns the cluster
 status.</p>
 <div class="codehilite"><pre><span class="k">if</span> <span class="n">live</span> <span class="n">and</span> <span class="n">not</span> <span class="n">slider</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">live</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">instancename</span><span class="p">):</span>
@@ -690,7 +701,7 @@ status.</p>
 </pre></div>
 
 
-<h2 id="action-getconf">Action: getConf</h2>
+<h2 id="action-getconf">Action: getConf<a class="headerlink" href="#action-getconf" title="Permanent link">&para;</a></h2>
 <p>This returns the live client configuration of the cluster -the
 site-xml file.</p>
 <div class="codehilite"><pre><span class="n">getconf</span> <span class="o">--</span><span class="n">format</span> <span class="p">(</span><span class="n">xml</span><span class="o">|</span><span class="k">properties</span><span class="p">)</span> <span class="o">--</span><span class="n">out</span> <span class="p">[</span><span class="n">outfile</span><span class="p">]</span>
@@ -698,12 +709,12 @@ site-xml file.</p>
 
 
 <p><em>We may want to think hard about whether this is needed</em></p>
-<h4 id="preconditions_10">Preconditions</h4>
+<h4 id="preconditions_10">Preconditions<a class="headerlink" href="#preconditions_10" title="Permanent link">&para;</a></h4>
 <div class="codehilite"><pre><span class="k">if</span> <span class="n">not</span> <span class="n">slider</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">live</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">instancename</span><span class="p">)</span> <span class="p">:</span> <span class="n">raise</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_UNKNOWN_INSTANCE</span><span class="p">)</span>
 </pre></div>
 
 
-<h4 id="postconditions_10">Postconditions</h4>
+<h4 id="postconditions_10">Postconditions<a class="headerlink" href="#postconditions_10" title="Permanent link">&para;</a></h4>
 <p>The operation succeeds if the cluster status can be retrieved and saved to 
 the named file/printed to stdout in the format chosen</p>
 <div class="codehilite"><pre><span class="n">let</span> <span class="n">status</span> <span class="p">=</span> <span class="n">slider</span><span class="o">-</span><span class="n">live</span><span class="o">-</span><span class="n">instances</span><span class="p">(</span><span class="n">YARN</span><span class="p">).</span><span class="n">rpcPort</span><span class="p">.</span><span class="n">getJSONClusterStatus</span><span class="p">()</span>
@@ -720,13 +731,13 @@ the named file/printed to stdout in the
 </pre></div>
 
 
-<h2 id="action-list">Action: list</h2>
+<h2 id="action-list">Action: list<a class="headerlink" href="#action-list" title="Permanent link">&para;</a></h2>
 <div class="codehilite"><pre><span class="n">list</span> <span class="p">[</span><span class="n">instancename</span><span class="p">]</span>
 </pre></div>
 
 
 <p>Lists all clusters of a user, or only the one given</p>
-<h4 id="preconditions_11">Preconditions</h4>
+<h4 id="preconditions_11">Preconditions<a class="headerlink" href="#preconditions_11" title="Permanent link">&para;</a></h4>
 <p>If a instancename is specified it must be in YARNs list of active or completed applications
 of that user:</p>
 <div class="codehilite"><pre><span class="k">if</span> <span class="n">instancename</span> !<span class="p">=</span> &quot;&quot; <span class="n">and</span> <span class="p">[]</span> <span class="o">==</span> <span class="n">yarn</span><span class="o">-</span><span class="n">application</span><span class="o">-</span><span class="n">instances</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">instancename</span><span class="p">,</span> <span class="n">user</span><span class="p">)</span> 
@@ -734,7 +745,7 @@ of that user:</p>
 </pre></div>
 
 
-<h4 id="postconditions_11">Postconditions</h4>
+<h4 id="postconditions_11">Postconditions<a class="headerlink" href="#postconditions_11" title="Permanent link">&para;</a></h4>
 <p>If no instancename was given, all slider applications of that user are listed,
 else only the one running (or one of the finished ones)</p>
 <div class="codehilite"><pre><span class="k">if</span> <span class="n">instancename</span> <span class="o">==</span> &quot;&quot; <span class="p">:</span>
@@ -746,7 +757,7 @@ else only the one running (or one of the
 </pre></div>
 
 
-<h2 id="action-killcontainer">Action: killcontainer</h2>
+<h2 id="action-killcontainer">Action: killcontainer<a class="headerlink" href="#action-killcontainer" title="Permanent link">&para;</a></h2>
 <p>This is an operation added for testing. It will kill a container in the cluster
 <em>without flexing the cluster size</em>. As a result, the cluster will detect the
 failure and attempt to recover from the failure by instantiating a new instance
@@ -755,7 +766,7 @@ of the cluster</p>
 </pre></div>
 
 
-<h4 id="preconditions_12">Preconditions</h4>
+<h4 id="preconditions_12">Preconditions<a class="headerlink" href="#preconditions_12" title="Permanent link">&para;</a></h4>
 <div class="codehilite"><pre><span class="k">if</span> <span class="n">not</span> <span class="n">slider</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">live</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">instancename</span><span class="p">)</span> <span class="p">:</span> <span class="n">raise</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_UNKNOWN_INSTANCE</span><span class="p">)</span>
 
 <span class="n">exists</span> <span class="n">c</span> <span class="n">in</span> <span class="n">slider</span><span class="o">-</span><span class="n">app</span><span class="o">-</span><span class="n">containers</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">instancename</span><span class="p">,</span> <span class="n">user</span><span class="p">)</span> <span class="n">where</span> <span class="n">c</span><span class="p">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">container</span><span class="o">-</span><span class="n">id</span>
@@ -765,7 +776,7 @@ of the cluster</p>
 </pre></div>
 
 
-<h4 id="postconditions_12">Postconditions</h4>
+<h4 id="postconditions_12">Postconditions<a class="headerlink" href="#postconditions_12" title="Permanent link">&para;</a></h4>
 <p>The container is not in the list of containers in the cluster</p>
 <div class="codehilite"><pre><span class="n">not</span> <span class="n">exists</span> <span class="n">c</span> <span class="n">in</span> <span class="n">containers</span><span class="p">(</span><span class="n">YARN</span><span class="p">)</span> <span class="n">where</span> <span class="n">c</span><span class="p">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">container</span><span class="o">-</span><span class="n">id</span>
 </pre></div>

Modified: websites/staging/slider/trunk/content/design/specification/index.html
==============================================================================
--- websites/staging/slider/trunk/content/design/specification/index.html (original)
+++ websites/staging/slider/trunk/content/design/specification/index.html Mon Nov  2 17:24:47 2015
@@ -155,7 +155,7 @@
   <div style="text-align: center">
     <h1><a href="/index.html">Apache Slider (incubating)</a></h1>
     <hr>
-Latest release: <strong>0.80.0-incubating</strong><br>
+Latest release: <strong>0.81.1-incubating</strong><br>
     <br>
     <a id="download-button-sidebar" class="btn btn-success btn-block" href="/downloads/" role="button">Download</a>
   </div>
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
 
     <h1 class="title"></h1>
 
-    <!---
+    <style type="text/css">
+/* The following code is added by mdx_elementid.py
+   It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+  visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink, h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink, dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
    Licensed to the Apache Software Foundation (ASF) under one or more
    contributor license agreements.  See the NOTICE file distributed with
    this work for additional information regarding copyright ownership.
@@ -185,7 +196,7 @@ Latest release: <strong>0.80.0-incubatin
    limitations under the License.
 -->
 
-<h1 id="specification-of-apache-slider-behaviour">Specification of Apache Slider behaviour</h1>
+<h1 id="specification-of-apache-slider-behaviour">Specification of Apache Slider behaviour<a class="headerlink" href="#specification-of-apache-slider-behaviour" title="Permanent link">&para;</a></h1>
 <p>This is a a "more rigorous" definition of the behavior of Slider in terms
 of its state and its command-line operations -by defining a 'formal' model
 of HDFS, YARN and Slider's internal state, then describing the operations

Modified: websites/staging/slider/trunk/content/design/specification/slider-model.html
==============================================================================
--- websites/staging/slider/trunk/content/design/specification/slider-model.html (original)
+++ websites/staging/slider/trunk/content/design/specification/slider-model.html Mon Nov  2 17:24:47 2015
@@ -155,7 +155,7 @@
   <div style="text-align: center">
     <h1><a href="/index.html">Apache Slider (incubating)</a></h1>
     <hr>
-Latest release: <strong>0.80.0-incubating</strong><br>
+Latest release: <strong>0.81.1-incubating</strong><br>
     <br>
     <a id="download-button-sidebar" class="btn btn-success btn-block" href="/downloads/" role="button">Download</a>
   </div>
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
 
     <h1 class="title"></h1>
 
-    <!---
+    <style type="text/css">
+/* The following code is added by mdx_elementid.py
+   It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+  visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink, h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink, dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
    Licensed to the Apache Software Foundation (ASF) under one or more
    contributor license agreements.  See the NOTICE file distributed with
    this work for additional information regarding copyright ownership.
@@ -185,9 +196,9 @@ Latest release: <strong>0.80.0-incubatin
    limitations under the License.
 -->
 
-<h1 id="formal-apache-slider-model">Formal Apache Slider Model</h1>
+<h1 id="formal-apache-slider-model">Formal Apache Slider Model<a class="headerlink" href="#formal-apache-slider-model" title="Permanent link">&para;</a></h1>
 <p>This is the model of Slider and YARN for the rest of the specification.</p>
-<h2 id="file-system">File System</h2>
+<h2 id="file-system">File System<a class="headerlink" href="#file-system" title="Permanent link">&para;</a></h2>
 <p>A File System <code>HDFS</code> represents a Hadoop FileSystem -either HDFS or another File
 System which spans the cluster. There are also other filesystems that
 can act as sources of data that is then copied into HDFS. These will be marked
@@ -201,7 +212,7 @@ Two key references are</p>
 <li><a href="https://github.com/steveloughran/hadoop-trunk/blob/stevel/HADOOP-9361-filesystem-contract/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/model.md">The model of the filesystem</a></li>
 </ol>
 <p>The model and its predicates and invariants will be used in these specifications.</p>
-<h2 id="yarn">YARN</h2>
+<h2 id="yarn">YARN<a class="headerlink" href="#yarn" title="Permanent link">&para;</a></h2>
 <p>From the perspective of YARN application, The YARN runtime is a state, <code>YARN</code>, 
 comprised of: <code>(Apps, Queues, Nodes)</code></p>
 <div class="codehilite"><pre><span class="n">Apps</span><span class="o">:</span> <span class="n">Map</span><span class="o">[</span><span class="n">AppId</span><span class="o">,</span> <span class="n">ApplicationReport</span><span class="o">]</span>
@@ -265,7 +276,7 @@ all nodes</p>
 </pre></div>
 
 
-<h3 id="operations-predicates-used-the-specifications">Operations &amp; predicates used the specifications</h3>
+<h3 id="operations-predicates-used-the-specifications">Operations &amp; predicates used the specifications<a class="headerlink" href="#operations-predicates-used-the-specifications" title="Permanent link">&para;</a></h3>
 <div class="codehilite"><pre><span class="n">def</span> <span class="n">applications</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">type</span><span class="p">)</span> <span class="p">=</span> 
     <span class="p">[</span> <span class="n">app</span><span class="p">.</span><span class="n">report</span> <span class="k">for</span> <span class="n">app</span> <span class="n">in</span> <span class="n">YARN</span><span class="p">.</span><span class="n">Apps</span><span class="p">.</span><span class="n">values</span> <span class="n">where</span> <span class="n">app</span><span class="p">.</span><span class="n">report</span><span class="p">.</span><span class="n">Type</span> <span class="o">==</span> <span class="n">type</span><span class="p">]</span>
 
@@ -274,7 +285,7 @@ all nodes</p>
 </pre></div>
 
 
-<h2 id="usergroupinformation">UserGroupInformation</h2>
+<h2 id="usergroupinformation">UserGroupInformation<a class="headerlink" href="#usergroupinformation" title="Permanent link">&para;</a></h2>
 <p>Applications are launched and executed on hosts computers: either client machines
 or nodes in the cluster, these have their own state which may need modeling</p>
 <div class="codehilite"><pre><span class="n">HostState</span><span class="o">:</span> <span class="n">Map</span><span class="o">[</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">]</span>
@@ -290,8 +301,8 @@ access to the filesystem and to parts of
 If it did they would be used throughout the specification to bind to a YARN or HDFS instance.</p>
 <p><code>UserGroupInformation.getCurrentUser(): UserGroupInformation</code></p>
 <p>Returns the current user information. This information is immutable and fixed for the duration of the process.</p>
-<h2 id="slider-model">Slider Model</h2>
-<h3 id="cluster-name">Cluster name</h3>
+<h2 id="slider-model">Slider Model<a class="headerlink" href="#slider-model" title="Permanent link">&para;</a></h2>
+<h3 id="cluster-name">Cluster name<a class="headerlink" href="#cluster-name" title="Permanent link">&para;</a></h3>
 <p>A valid cluster name is a name of length &gt; 1 which follows the internet hostname scheme of letter followed by letter or digit</p>
 <div class="codehilite"><pre><span class="n">def</span> <span class="n">valid</span><span class="o">-</span><span class="n">cluster</span><span class="o">-</span><span class="n">name</span><span class="p">(</span><span class="n">c</span><span class="p">)</span> <span class="p">=</span>
     <span class="n">len</span><span class="p">(</span><span class="n">c</span><span class="p">)</span><span class="o">&gt;</span> 0
@@ -300,7 +311,7 @@ If it did they would be used throughout
 </pre></div>
 
 
-<h3 id="persistent-cluster-state">Persistent Cluster State</h3>
+<h3 id="persistent-cluster-state">Persistent Cluster State<a class="headerlink" href="#persistent-cluster-state" title="Permanent link">&para;</a></h3>
 <p>A Slider cluster's persistent state is stored in a path</p>
 <div class="codehilite"><pre><span class="n">def</span> <span class="n">cluster</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">clustername</span><span class="p">)</span> <span class="p">=</span> <span class="n">user</span><span class="o">-</span><span class="n">home</span><span class="p">(</span><span class="n">FS</span><span class="p">)</span> <span class="o">+</span> <span class="p">[</span>&quot;<span class="n">clusters</span>&quot;<span class="p">,</span> <span class="n">clustername</span><span class="p">]</span>
 <span class="n">def</span> <span class="n">cluster</span><span class="o">-</span><span class="n">json</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">clustername</span><span class="p">)</span> <span class="p">=</span> <span class="n">cluster</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">clustername</span><span class="p">)</span> <span class="o">+</span> <span class="p">[</span>&quot;<span class="n">cluster</span><span class="p">.</span><span class="n">json</span>&quot;<span class="p">]</span>
@@ -346,7 +357,7 @@ specific instance bindings and saved int
 </pre></div>
 
 
-<h3 id="invariant-there-must-never-be-more-than-one-running-instance-of-a-named-slider-cluster">Invariant: there must never be more than one running instance of a named Slider cluster</h3>
+<h3 id="invariant-there-must-never-be-more-than-one-running-instance-of-a-named-slider-cluster">Invariant: there must never be more than one running instance of a named Slider cluster<a class="headerlink" href="#invariant-there-must-never-be-more-than-one-running-instance-of-a-named-slider-cluster" title="Permanent link">&para;</a></h3>
 <p>There must never be more than one instance of the same Slider cluster running:</p>
 <div class="codehilite"><pre><span class="n">forall</span> <span class="n">a</span> <span class="n">in</span> <span class="n">user</span><span class="o">-</span><span class="n">applications</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> &quot;<span class="n">slider</span>&quot;<span class="p">,</span> <span class="n">user</span><span class="p">):</span>
     <span class="n">len</span><span class="p">(</span><span class="n">slider</span><span class="o">-</span><span class="n">app</span><span class="o">-</span><span class="n">running</span><span class="o">-</span><span class="n">instances</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">a</span><span class="p">.</span><span class="n">Name</span><span class="p">,</span> <span class="n">user</span><span class="p">))</span> <span class="o">&lt;</span><span class="p">=</span> 1
@@ -355,7 +366,7 @@ specific instance bindings and saved int
 
 <p>There may be multiple instances in a finished state, and one running instance alongside multiple finished instances -the applications
 that work with Slider MUST select a running cluster ahead of any terminated clusters.</p>
-<h3 id="containers-of-an-application">Containers of an application</h3>
+<h3 id="containers-of-an-application">Containers of an application<a class="headerlink" href="#containers-of-an-application" title="Permanent link">&para;</a></h3>
 <p>The containers of a slider application are the set of containers of that application</p>
 <div class="codehilite"><pre><span class="n">def</span> <span class="n">slider</span><span class="o">-</span><span class="n">app</span><span class="o">-</span><span class="n">containers</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">clustername</span><span class="p">,</span> <span class="n">user</span><span class="p">)</span> <span class="p">=</span>
   <span class="n">app</span><span class="o">-</span><span class="n">containers</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">appid</span> <span class="n">where</span>
@@ -363,7 +374,7 @@ that work with Slider MUST select a runn
 </pre></div>
 
 
-<h3 id="rpc-access-to-a-slider-cluster">RPC Access to a slider cluster</h3>
+<h3 id="rpc-access-to-a-slider-cluster">RPC Access to a slider cluster<a class="headerlink" href="#rpc-access-to-a-slider-cluster" title="Permanent link">&para;</a></h3>
 <p>An application is accepting RPC requests for a given protocol if there is a port binding
  defined and it is possible to authenticate a connection using the specified protocol</p>
 <div class="codehilite"><pre> <span class="n">def</span> <span class="n">rpc</span><span class="o">-</span><span class="n">connection</span><span class="p">(</span><span class="n">appReport</span><span class="p">,</span> <span class="n">protocol</span><span class="p">)</span> <span class="p">=</span>
@@ -375,9 +386,9 @@ that work with Slider MUST select a runn
 
 <p>Being able to open an RPC port is the strongest definition of liveness possible
  to make: if the AM responds to RPC operations, it is doing useful work.</p>
-<h3 id="valid-cluster-description">Valid Cluster Description</h3>
+<h3 id="valid-cluster-description">Valid Cluster Description<a class="headerlink" href="#valid-cluster-description" title="Permanent link">&para;</a></h3>
 <p>The <code>cluster.json</code> file of a cluster configures Slider to deploy the application. </p>
-<h4 id="well-defined-clustercluster-description">well-defined-cluster(cluster-description)</h4>
+<h4 id="well-defined-clustercluster-description">well-defined-cluster(cluster-description)<a class="headerlink" href="#well-defined-clustercluster-description" title="Permanent link">&para;</a></h4>
 <p>A Cluster Description is well-defined if it is valid JSON and required properties are present</p>
 <p><strong>OBSOLETE</strong></p>
 <p>Irrespective of specific details for deploying the Slider AM or any provider-specific role instances,
@@ -396,7 +407,7 @@ is well-defined if</p>
 <code>org.apache.slider.api.ClusterDescription</code> </p>
 <p>Currently Slider ignores unknown elements during parsing. This may be changed.</p>
 <p>The test for this state does not refer to the cluster filesystem</p>
-<h4 id="deployable-clusterfs-cluster-description">deployable-cluster(FS, cluster-description)</h4>
+<h4 id="deployable-clusterfs-cluster-description">deployable-cluster(FS, cluster-description)<a class="headerlink" href="#deployable-clusterfs-cluster-description" title="Permanent link">&para;</a></h4>
 <p>A  Cluster Description defines a deployable cluster if it is well-defined cluster and the contents contain valid information to deploy a cluster</p>
 <p>This defines how a cluster description is valid in the extends the valid configuration with </p>
 <ul>
@@ -434,7 +445,7 @@ and cast to <code>SliderProviderFactory<
 
 </li>
 </ul>
-<h4 id="valid-for-providercluster-description-provider">valid-for-provider(cluster-description, provider)</h4>
+<h4 id="valid-for-providercluster-description-provider">valid-for-provider(cluster-description, provider)<a class="headerlink" href="#valid-for-providercluster-description-provider" title="Permanent link">&para;</a></h4>
 <p>A provider considers a specification valid if its own validation logic is satisfied. This normally
 consists of rules about the number of instances of different roles; it may include other logic.</p>
   </div>