You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@slider.apache.org by sm...@apache.org on 2014/07/08 02:49:35 UTC
svn commit: r1608633 [2/7] - in /incubator/slider/site: content/design/
content/design/registry/ content/design/specification/ content/developing/
content/docs/ content/docs/configuration/ content/docs/slider_specs/
trunk/content/developing/
Modified: incubator/slider/site/content/design/specification/cli-actions.html
URL: http://svn.apache.org/viewvc/incubator/slider/site/content/design/specification/cli-actions.html?rev=1608633&r1=1608632&r2=1608633&view=diff
==============================================================================
--- incubator/slider/site/content/design/specification/cli-actions.html (original)
+++ incubator/slider/site/content/design/specification/cli-actions.html Tue Jul 8 00:49:34 2014
@@ -195,41 +195,46 @@ if one is not provided.</p>
<h2 id="action-build">Action: Build</h2>
<p>Builds a cluster -creates all the on-filesystem datastructures, and generates a cluster description
that is both well-defined and deployable -<em>but does not actually start the cluster</em></p>
-<pre class="codehilite"><code>build (instancename,
- options:List[(String,String)],
- components:List[(String, int)],
- componentOptions:List[(String,String, String)],
- resourceOptions:List[(String,String)],
- resourceComponentOptions:List[(String,String, String)],
- confdir: URI,
- provider: String
- zkhosts,
- zkport,
- image
- apphome
- appconfdir</code></pre>
+<div class="codehilite"><pre><span class="n">build</span> <span class="p">(</span><span class="n">instancename</span><span class="p">,</span>
+ <span class="nl">options:</span><span class="n">List</span><span class="p">[(</span><span class="n">String</span><span class="p">,</span><span class="n">String</span><span class="p">)],</span>
+ <span class="nl">components:</span><span class="n">List</span><span class="p">[(</span><span class="n">String</span><span class="p">,</span> <span class="kt">int</span><span class="p">)],</span>
+ <span class="nl">componentOptions:</span><span class="n">List</span><span class="p">[(</span><span class="n">String</span><span class="p">,</span><span class="n">String</span><span class="p">,</span> <span class="n">String</span><span class="p">)],</span>
+ <span class="nl">resourceOptions:</span><span class="n">List</span><span class="p">[(</span><span class="n">String</span><span class="p">,</span><span class="n">String</span><span class="p">)],</span>
+ <span class="nl">resourceComponentOptions:</span><span class="n">List</span><span class="p">[(</span><span class="n">String</span><span class="p">,</span><span class="n">String</span><span class="p">,</span> <span class="n">String</span><span class="p">)],</span>
+ <span class="nl">confdir:</span> <span class="n">URI</span><span class="p">,</span>
+ <span class="nl">provider:</span> <span class="n">String</span>
+ <span class="n">zkhosts</span><span class="p">,</span>
+ <span class="n">zkport</span><span class="p">,</span>
+ <span class="n">image</span>
+ <span class="n">apphome</span>
+ <span class="n">appconfdir</span>
+</pre></div>
<h4 id="preconditions">Preconditions</h4>
<p>(Note that the ordering of these preconditions is not guaranteed to remain constant)</p>
<p>The instance name is valid</p>
-<pre class="codehilite"><code>if not valid-instance-name(instancename) : raise SliderException(EXIT_COMMAND_ARGUMENT_ERROR)</code></pre>
+<div class="codehilite"><pre><span class="k">if</span> <span class="n">not</span> <span class="n">valid</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">name</span><span class="p">(</span><span class="n">instancename</span><span class="p">)</span> <span class="o">:</span> <span class="n">raise</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_COMMAND_ARGUMENT_ERROR</span><span class="p">)</span>
+</pre></div>
<p>The instance must not be live. This is purely a safety check as the next test should have the same effect.</p>
-<pre class="codehilite"><code>if slider-instance-live(YARN, instancename) : raise SliderException(EXIT_CLUSTER_IN_USE)</code></pre>
+<div class="codehilite"><pre><span class="k">if</span> <span class="n">slider</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">live</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">instancename</span><span class="p">)</span> <span class="o">:</span> <span class="n">raise</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_CLUSTER_IN_USE</span><span class="p">)</span>
+</pre></div>
<p>The instance must not exist</p>
-<pre class="codehilite"><code>if is-dir(HDFS, instance-path(FS, instancename)) : raise SliderException(EXIT_CLUSTER_EXISTS)</code></pre>
+<div class="codehilite"><pre><span class="k">if</span> <span class="n">is</span><span class="o">-</span><span class="n">dir</span><span class="p">(</span><span class="n">HDFS</span><span class="p">,</span> <span class="n">instance</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">instancename</span><span class="p">))</span> <span class="o">:</span> <span class="n">raise</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_CLUSTER_EXISTS</span><span class="p">)</span>
+</pre></div>
<p>The configuration directory must exist it does not have to be the instance's HDFS instance,
as it will be copied there -and must contain only files</p>
-<pre class="codehilite"><code>let FS = FileSystem.get(appconfdir)
-if not isDir(FS, appconfdir) raise SliderException(EXIT_COMMAND_ARGUMENT_ERROR)
-forall f in children(FS, appconfdir) :
- if not isFile(f): raise IOException</code></pre>
+<div class="codehilite"><pre><span class="n">let</span> <span class="n">FS</span> <span class="o">=</span> <span class="n">FileSystem</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">appconfdir</span><span class="p">)</span>
+<span class="k">if</span> <span class="n">not</span> <span class="n">isDir</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">appconfdir</span><span class="p">)</span> <span class="n">raise</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_COMMAND_ARGUMENT_ERROR</span><span class="p">)</span>
+<span class="n">forall</span> <span class="n">f</span> <span class="n">in</span> <span class="n">children</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">appconfdir</span><span class="p">)</span> <span class="o">:</span>
+ <span class="k">if</span> <span class="n">not</span> <span class="n">isFile</span><span class="p">(</span><span class="n">f</span><span class="p">)</span><span class="o">:</span> <span class="n">raise</span> <span class="n">IOException</span>
+</pre></div>
<p>There's a race condition at build time where between the preconditions being met and the instance specification being saved, the instance
@@ -242,68 +247,76 @@ writelock file, failing if its present.
when a process successfully acquires the write lock. If this proves to be an issue, a stricter model could be implemented, with each reading process creating a unique named readlock- file.</p>
<h4 id="postconditions">Postconditions</h4>
<p>All the instance directories exist</p>
-<pre class="codehilite"><code>is-dir(HDFS', instance-path(HDFS', instancename))
-is-dir(HDFS', original-conf-path(HDFS', instancename))
-is-dir(HDFS', generated-conf-path(HDFS', instancename))</code></pre>
+<div class="codehilite"><pre><span class="n">is</span><span class="o">-</span><span class="n">dir</span><span class="p">(</span><span class="n">HDFS</span><span class="err">'</span><span class="p">,</span> <span class="n">instance</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">HDFS</span><span class="err">'</span><span class="p">,</span> <span class="n">instancename</span><span class="p">))</span>
+<span class="n">is</span><span class="o">-</span><span class="n">dir</span><span class="p">(</span><span class="n">HDFS</span><span class="err">'</span><span class="p">,</span> <span class="n">original</span><span class="o">-</span><span class="n">conf</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">HDFS</span><span class="err">'</span><span class="p">,</span> <span class="n">instancename</span><span class="p">))</span>
+<span class="n">is</span><span class="o">-</span><span class="n">dir</span><span class="p">(</span><span class="n">HDFS</span><span class="err">'</span><span class="p">,</span> <span class="n">generated</span><span class="o">-</span><span class="n">conf</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">HDFS</span><span class="err">'</span><span class="p">,</span> <span class="n">instancename</span><span class="p">))</span>
+</pre></div>
<p>The application cluster specification saved is well-defined and deployable</p>
-<pre class="codehilite"><code>let instance-description = parse(data(HDFS', instance-json-path(HDFS', instancename)))
-well-defined-instance(instance-description)
-deployable-application-instance(HDFS', instance-description)</code></pre>
+<div class="codehilite"><pre><span class="n">let</span> <span class="n">instance</span><span class="o">-</span><span class="n">description</span> <span class="o">=</span> <span class="n">parse</span><span class="p">(</span><span class="n">data</span><span class="p">(</span><span class="n">HDFS</span><span class="err">'</span><span class="p">,</span> <span class="n">instance</span><span class="o">-</span><span class="n">json</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">HDFS</span><span class="err">'</span><span class="p">,</span> <span class="n">instancename</span><span class="p">)))</span>
+<span class="n">well</span><span class="o">-</span><span class="n">defined</span><span class="o">-</span><span class="n">instance</span><span class="p">(</span><span class="n">instance</span><span class="o">-</span><span class="n">description</span><span class="p">)</span>
+<span class="n">deployable</span><span class="o">-</span><span class="n">application</span><span class="o">-</span><span class="n">instance</span><span class="p">(</span><span class="n">HDFS</span><span class="err">'</span><span class="p">,</span> <span class="n">instance</span><span class="o">-</span><span class="n">description</span><span class="p">)</span>
+</pre></div>
<p>More precisely: the specification generated before it is saved as JSON is well-defined and deployable; no JSON file will be created
if the validation fails.</p>
<p>Fields in the cluster description have been filled in</p>
-<pre class="codehilite"><code>internal.global["internal.provider.name"] == provider
-app_conf.global["zookeeper.port"] == zkport
-app_conf.global["zookeeper.hosts"] == zkhosts
+<div class="codehilite"><pre><span class="n">internal</span><span class="p">.</span><span class="n">global</span><span class="p">[</span><span class="s">"internal.provider.name"</span><span class="p">]</span> <span class="o">==</span> <span class="n">provider</span>
+<span class="n">app_conf</span><span class="p">.</span><span class="n">global</span><span class="p">[</span><span class="s">"zookeeper.port"</span><span class="p">]</span> <span class="o">==</span> <span class="n">zkport</span>
+<span class="n">app_conf</span><span class="p">.</span><span class="n">global</span><span class="p">[</span><span class="s">"zookeeper.hosts"</span><span class="p">]</span> <span class="o">==</span> <span class="n">zkhosts</span>
-package => app_conf.global["agent.package"] = package</code></pre>
+<span class="n">package</span> <span class="o">=></span> <span class="n">app_conf</span><span class="p">.</span><span class="n">global</span><span class="p">[</span><span class="s">"agent.package"</span><span class="p">]</span> <span class="o">=</span> <span class="n">package</span>
+</pre></div>
<p>Any <code>apphome</code> and <code>image</code> properties have propagated</p>
-<pre class="codehilite"><code>apphome == null or clusterspec.options["cluster.application.home"] == apphome
-image == null or clusterspec.options["cluster.application.image.path"] == image</code></pre>
+<div class="codehilite"><pre><span class="n">apphome</span> <span class="o">==</span> <span class="n">null</span> <span class="n">or</span> <span class="n">clusterspec</span><span class="p">.</span><span class="n">options</span><span class="p">[</span><span class="s">"cluster.application.home"</span><span class="p">]</span> <span class="o">==</span> <span class="n">apphome</span>
+<span class="n">image</span> <span class="o">==</span> <span class="n">null</span> <span class="n">or</span> <span class="n">clusterspec</span><span class="p">.</span><span class="n">options</span><span class="p">[</span><span class="s">"cluster.application.image.path"</span><span class="p">]</span> <span class="o">==</span> <span class="n">image</span>
+</pre></div>
<p>(The <code>well-defined-application-instance()</code> requirement above defines the valid states
of this pair of options)</p>
<p>All role sizes have been mapped to <code>component.instances</code> fields</p>
-<pre class="codehilite"><code>forall (name, size) in components :
- resources.components[name]["components.instances"] == size</code></pre>
+<div class="codehilite"><pre><span class="n">forall</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">size</span><span class="p">)</span> <span class="n">in</span> <span class="n">components</span> <span class="o">:</span>
+ <span class="n">resources</span><span class="p">.</span><span class="n">components</span><span class="p">[</span><span class="n">name</span><span class="p">][</span><span class="s">"components.instances"</span><span class="p">]</span> <span class="o">==</span> <span class="n">size</span>
+</pre></div>
<p>All option parameters have been added to the <code>options</code> map in the specification</p>
-<pre class="codehilite"><code>forall (opt, val) in options :
- app_conf.global[opt] == val
+<div class="codehilite"><pre><span class="n">forall</span> <span class="p">(</span><span class="n">opt</span><span class="p">,</span> <span class="n">val</span><span class="p">)</span> <span class="n">in</span> <span class="n">options</span> <span class="o">:</span>
+ <span class="n">app_conf</span><span class="p">.</span><span class="n">global</span><span class="p">[</span><span class="n">opt</span><span class="p">]</span> <span class="o">==</span> <span class="n">val</span>
-forall (opt, val) in resourceOptions :
- resource.global[opt] == val</code></pre>
+<span class="n">forall</span> <span class="p">(</span><span class="n">opt</span><span class="p">,</span> <span class="n">val</span><span class="p">)</span> <span class="n">in</span> <span class="n">resourceOptions</span> <span class="o">:</span>
+ <span class="n">resource</span><span class="p">.</span><span class="n">global</span><span class="p">[</span><span class="n">opt</span><span class="p">]</span> <span class="o">==</span> <span class="n">val</span>
+</pre></div>
<p>All component option parameters have been added to the specific components's option map
in the relevant configuration file</p>
-<pre class="codehilite"><code>forall (name, opt, val) in componentOptions :
- app_conf.components[name][opt] == val
+<div class="codehilite"><pre><span class="n">forall</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">opt</span><span class="p">,</span> <span class="n">val</span><span class="p">)</span> <span class="n">in</span> <span class="n">componentOptions</span> <span class="o">:</span>
+ <span class="n">app_conf</span><span class="p">.</span><span class="n">components</span><span class="p">[</span><span class="n">name</span><span class="p">][</span><span class="n">opt</span><span class="p">]</span> <span class="o">==</span> <span class="n">val</span>
-forall (name, opt, val) in resourceComponentOptions :
- resourceComponentOptions.components[name][opt] == val</code></pre>
+<span class="n">forall</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">opt</span><span class="p">,</span> <span class="n">val</span><span class="p">)</span> <span class="n">in</span> <span class="n">resourceComponentOptions</span> <span class="o">:</span>
+ <span class="n">resourceComponentOptions</span><span class="p">.</span><span class="n">components</span><span class="p">[</span><span class="n">name</span><span class="p">][</span><span class="n">opt</span><span class="p">]</span> <span class="o">==</span> <span class="n">val</span>
+</pre></div>
<p>To avoid some confusion as to where keys go, all options beginning with the
prefix <code>component.</code> are automatically copied into the resources file:</p>
-<pre class="codehilite"><code>forall (opt, val) in options where startswith(opt, "component.")
- or startswith(opt, "role.")
- or startswith(opt, "yarn."):
- resource.global[opt] == val
-
-forall (name, opt, val) in componentOptions where startswith(opt, "component.")
- or startswith(opt, "role.")
- or startswith(opt, "yarn."):
- resourceComponentOptions.components[name][opt] == val</code></pre>
+<div class="codehilite"><pre><span class="n">forall</span> <span class="p">(</span><span class="n">opt</span><span class="p">,</span> <span class="n">val</span><span class="p">)</span> <span class="n">in</span> <span class="n">options</span> <span class="n">where</span> <span class="n">startswith</span><span class="p">(</span><span class="n">opt</span><span class="p">,</span> <span class="s">"component."</span><span class="p">)</span>
+ <span class="n">or</span> <span class="n">startswith</span><span class="p">(</span><span class="n">opt</span><span class="p">,</span> <span class="s">"role."</span><span class="p">)</span>
+ <span class="n">or</span> <span class="n">startswith</span><span class="p">(</span><span class="n">opt</span><span class="p">,</span> <span class="s">"yarn."</span><span class="p">)</span><span class="o">:</span>
+ <span class="n">resource</span><span class="p">.</span><span class="n">global</span><span class="p">[</span><span class="n">opt</span><span class="p">]</span> <span class="o">==</span> <span class="n">val</span>
+
+<span class="n">forall</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">opt</span><span class="p">,</span> <span class="n">val</span><span class="p">)</span> <span class="n">in</span> <span class="n">componentOptions</span> <span class="n">where</span> <span class="n">startswith</span><span class="p">(</span><span class="n">opt</span><span class="p">,</span> <span class="s">"component."</span><span class="p">)</span>
+ <span class="n">or</span> <span class="n">startswith</span><span class="p">(</span><span class="n">opt</span><span class="p">,</span> <span class="s">"role."</span><span class="p">)</span>
+ <span class="n">or</span> <span class="n">startswith</span><span class="p">(</span><span class="n">opt</span><span class="p">,</span> <span class="s">"yarn."</span><span class="p">)</span><span class="o">:</span>
+ <span class="n">resourceComponentOptions</span><span class="p">.</span><span class="n">components</span><span class="p">[</span><span class="n">name</span><span class="p">][</span><span class="n">opt</span><span class="p">]</span> <span class="o">==</span> <span class="n">val</span>
+</pre></div>
<p>There's no explicit rejection of duplicate options, the outcome of that
@@ -311,54 +324,63 @@ state is 'undefined'. </p>
<p>What is defined is that if Slider or its provider provided a default option value,
the command-line supplied option will override it.</p>
<p>All files that were in the configuration directory are now copied into the "original" configuration directory</p>
-<pre class="codehilite"><code>let FS = FileSystem.get(appconfdir)
-let dest = original-conf-path(HDFS', instancename)
-forall [c in children(FS, confdir) :
- data(HDFS', dest + [filename(c)]) == data(FS, c)</code></pre>
+<div class="codehilite"><pre><span class="n">let</span> <span class="n">FS</span> <span class="o">=</span> <span class="n">FileSystem</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">appconfdir</span><span class="p">)</span>
+<span class="n">let</span> <span class="n">dest</span> <span class="o">=</span> <span class="n">original</span><span class="o">-</span><span class="n">conf</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">HDFS</span><span class="err">'</span><span class="p">,</span> <span class="n">instancename</span><span class="p">)</span>
+<span class="n">forall</span> <span class="p">[</span><span class="n">c</span> <span class="n">in</span> <span class="n">children</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">confdir</span><span class="p">)</span> <span class="o">:</span>
+ <span class="n">data</span><span class="p">(</span><span class="n">HDFS</span><span class="err">'</span><span class="p">,</span> <span class="n">dest</span> <span class="o">+</span> <span class="p">[</span><span class="n">filename</span><span class="p">(</span><span class="n">c</span><span class="p">)])</span> <span class="o">==</span> <span class="n">data</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">c</span><span class="p">)</span>
+</pre></div>
<p>All files that were in the configuration directory now have equivalents in the generated configuration directory</p>
-<pre class="codehilite"><code>let FS = FileSystem.get(appconfdir)
-let dest = generated-conf-path(HDFS', instancename)
-forall [c in children(FS, confdir) :
- isfile(HDFS', dest + [filename(c)])</code></pre>
+<div class="codehilite"><pre><span class="n">let</span> <span class="n">FS</span> <span class="o">=</span> <span class="n">FileSystem</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">appconfdir</span><span class="p">)</span>
+<span class="n">let</span> <span class="n">dest</span> <span class="o">=</span> <span class="n">generated</span><span class="o">-</span><span class="n">conf</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">HDFS</span><span class="err">'</span><span class="p">,</span> <span class="n">instancename</span><span class="p">)</span>
+<span class="n">forall</span> <span class="p">[</span><span class="n">c</span> <span class="n">in</span> <span class="n">children</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">confdir</span><span class="p">)</span> <span class="o">:</span>
+ <span class="n">isfile</span><span class="p">(</span><span class="n">HDFS</span><span class="err">'</span><span class="p">,</span> <span class="n">dest</span> <span class="o">+</span> <span class="p">[</span><span class="n">filename</span><span class="p">(</span><span class="n">c</span><span class="p">)])</span>
+</pre></div>
<h2 id="action-thaw">Action: Thaw</h2>
-<pre class="codehilite"><code>thaw <instancename> [--wait <timeout>]</code></pre>
+<div class="codehilite"><pre><span class="nx">thaw</span> <span class="o"><</span><span class="nx">instancename</span><span class="o">></span> <span class="err">[</span><span class="o">--</span><span class="nb">wait</span> <span class="o"><</span><span class="nx">timeout</span><span class="o">></span><span class="cp">]</span>
+</pre></div>
<p>Thaw takes an application instance with configuration and (possibly) data on disk, and
attempts to create a live application with the specified number of nodes</p>
<h4 id="preconditions_1">Preconditions</h4>
-<pre class="codehilite"><code>if not valid-instance-name(instancename) : raise SliderException(EXIT_COMMAND_ARGUMENT_ERROR)</code></pre>
+<div class="codehilite"><pre><span class="k">if</span> <span class="n">not</span> <span class="n">valid</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">name</span><span class="p">(</span><span class="n">instancename</span><span class="p">)</span> <span class="o">:</span> <span class="n">raise</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_COMMAND_ARGUMENT_ERROR</span><span class="p">)</span>
+</pre></div>
<p>The cluster must not be live. This is purely a safety check as the next test should have the same effect.</p>
-<pre class="codehilite"><code>if slider-instance-live(YARN, instancename) : raise SliderException(EXIT_CLUSTER_IN_USE)</code></pre>
+<div class="codehilite"><pre><span class="k">if</span> <span class="n">slider</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">live</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">instancename</span><span class="p">)</span> <span class="o">:</span> <span class="n">raise</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_CLUSTER_IN_USE</span><span class="p">)</span>
+</pre></div>
<p>The cluster must not exist</p>
-<pre class="codehilite"><code>if is-dir(HDFS, application-instance-path(FS, instancename)) : raise SliderException(EXIT_CLUSTER_EXISTS)</code></pre>
+<div class="codehilite"><pre><span class="k">if</span> <span class="n">is</span><span class="o">-</span><span class="n">dir</span><span class="p">(</span><span class="n">HDFS</span><span class="p">,</span> <span class="n">application</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">instancename</span><span class="p">))</span> <span class="o">:</span> <span class="n">raise</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_CLUSTER_EXISTS</span><span class="p">)</span>
+</pre></div>
<p>The cluster specification must exist, be valid and deployable</p>
-<pre class="codehilite"><code>if not is-file(HDFS, cluster-json-path(HDFS, instancename)) : SliderException(EXIT_UNKNOWN_INSTANCE)
-if not well-defined-application-instance(HDFS, application-instance-path(HDFS, instancename)) : raise SliderException(EXIT_BAD_CLUSTER_STATE)
-if not deployable-application-instance(HDFS, application-instance-path(HDFS, instancename)) : raise SliderException(EXIT_BAD_CLUSTER_STATE)</code></pre>
+<div class="codehilite"><pre><span class="k">if</span> <span class="n">not</span> <span class="n">is</span><span class="o">-</span><span class="n">file</span><span class="p">(</span><span class="n">HDFS</span><span class="p">,</span> <span class="n">cluster</span><span class="o">-</span><span class="n">json</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">HDFS</span><span class="p">,</span> <span class="n">instancename</span><span class="p">))</span> <span class="o">:</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_UNKNOWN_INSTANCE</span><span class="p">)</span>
+<span class="k">if</span> <span class="n">not</span> <span class="n">well</span><span class="o">-</span><span class="n">defined</span><span class="o">-</span><span class="n">application</span><span class="o">-</span><span class="n">instance</span><span class="p">(</span><span class="n">HDFS</span><span class="p">,</span> <span class="n">application</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">HDFS</span><span class="p">,</span> <span class="n">instancename</span><span class="p">))</span> <span class="o">:</span> <span class="n">raise</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_BAD_CLUSTER_STATE</span><span class="p">)</span>
+<span class="k">if</span> <span class="n">not</span> <span class="n">deployable</span><span class="o">-</span><span class="n">application</span><span class="o">-</span><span class="n">instance</span><span class="p">(</span><span class="n">HDFS</span><span class="p">,</span> <span class="n">application</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">HDFS</span><span class="p">,</span> <span class="n">instancename</span><span class="p">))</span> <span class="o">:</span> <span class="n">raise</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_BAD_CLUSTER_STATE</span><span class="p">)</span>
+</pre></div>
<h3 id="postconditions_1">Postconditions</h3>
<p>After the thaw has been performed, there is now a queued request in YARN
for the chosen (how?) queue</p>
-<pre class="codehilite"><code>YARN'.Queues'[amqueue] = YARN.Queues[amqueue] + [launch("slider", instancename, requirements, context)]</code></pre>
+<div class="codehilite"><pre><span class="n">YARN</span><span class="err">'</span><span class="p">.</span><span class="n">Queues</span><span class="err">'</span><span class="p">[</span><span class="n">amqueue</span><span class="p">]</span> <span class="o">=</span> <span class="n">YARN</span><span class="p">.</span><span class="n">Queues</span><span class="p">[</span><span class="n">amqueue</span><span class="p">]</span> <span class="o">+</span> <span class="p">[</span><span class="n">launch</span><span class="p">(</span><span class="s">"slider"</span><span class="p">,</span> <span class="n">instancename</span><span class="p">,</span> <span class="n">requirements</span><span class="p">,</span> <span class="n">context</span><span class="p">)]</span>
+</pre></div>
<p>If a wait timeout was specified, the cli waits until the application is considered
running by YARN (the AM is running), the wait timeout has been reached, or
the application has failed</p>
-<pre class="codehilite"><code>waittime < 0 or (exists a in slider-running-application-instances(yarn-application-instances(YARN', instancename, user))
- where a.YarnApplicationState == RUNNING)</code></pre>
+<div class="codehilite"><pre><span class="n">waittime</span> <span class="o"><</span> <span class="mi">0</span> <span class="n">or</span> <span class="p">(</span><span class="n">exists</span> <span class="n">a</span> <span class="n">in</span> <span class="n">slider</span><span class="o">-</span><span class="n">running</span><span class="o">-</span><span class="n">application</span><span class="o">-</span><span class="n">instances</span><span class="p">(</span><span class="n">yarn</span><span class="o">-</span><span class="n">application</span><span class="o">-</span><span class="n">instances</span><span class="p">(</span><span class="n">YARN</span><span class="err">'</span><span class="p">,</span> <span class="n">instancename</span><span class="p">,</span> <span class="n">user</span><span class="p">))</span>
+ <span class="n">where</span> <span class="n">a</span><span class="p">.</span><span class="n">YarnApplicationState</span> <span class="o">==</span> <span class="n">RUNNING</span><span class="p">)</span>
+</pre></div>
<h2 id="outcome-am-launched-state">Outcome: AM-launched state</h2>
@@ -376,12 +398,14 @@ specific YARN application queue.</li>
an AM is launched afterwards</p>
<p>The AM is deployed if there is some time <code>t</code> after the submission time <code>t0</code>
where the application is listed </p>
-<pre class="codehilite"><code>exists t1 where t1 > t0 and slider-instance-live(YARN(t1), user, instancename)</code></pre>
+<div class="codehilite"><pre><span class="n">exists</span> <span class="n">t1</span> <span class="n">where</span> <span class="n">t1</span> <span class="o">></span> <span class="n">t0</span> <span class="n">and</span> <span class="n">slider</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">live</span><span class="p">(</span><span class="n">YARN</span><span class="p">(</span><span class="n">t1</span><span class="p">),</span> <span class="n">user</span><span class="p">,</span> <span class="n">instancename</span><span class="p">)</span>
+</pre></div>
<p>At which time there is a container in the cluster hosting the AM -it's
context is the launch context</p>
-<pre class="codehilite"><code>exists c in containers(YARN(t1)) where container.context = launch.context</code></pre>
+<div class="codehilite"><pre><span class="n">exists</span> <span class="n">c</span> <span class="n">in</span> <span class="n">containers</span><span class="p">(</span><span class="n">YARN</span><span class="p">(</span><span class="n">t1</span><span class="p">))</span> <span class="n">where</span> <span class="n">container</span><span class="p">.</span><span class="n">context</span> <span class="o">=</span> <span class="n">launch</span><span class="p">.</span><span class="n">context</span>
+</pre></div>
<p>There's no way to determine when this time <code>t1</code> will be reached -or if it ever
@@ -407,7 +431,8 @@ with an error code.</li>
will trigger a restart attempt -this may be on the same or a different node.</p>
<h4 id="preconditions_3">preconditions</h4>
<p>The AM was launched at an earlier time, <code>t1</code></p>
-<pre class="codehilite"><code>exists t1 where t1 > t0 and am-launched(YARN(t1)</code></pre>
+<div class="codehilite"><pre><span class="n">exists</span> <span class="n">t1</span> <span class="n">where</span> <span class="n">t1</span> <span class="o">></span> <span class="n">t0</span> <span class="n">and</span> <span class="n">am</span><span class="o">-</span><span class="n">launched</span><span class="p">(</span><span class="n">YARN</span><span class="p">(</span><span class="n">t1</span><span class="p">)</span>
+</pre></div>
<h4 id="postconditions_3">Postconditions</h4>
@@ -415,11 +440,12 @@ will trigger a restart attempt -this may
as being in the state <code>RUNNING</code>, an RPC port has been registered with YARN (visible as the <code>rpcPort</code>
attribute in the YARN Application Report,and that port is servicing RPC requests
from authenticated callers.</p>
-<pre class="codehilite"><code>exists t2 where:
- t2 > t1
- and slider-instance-live(YARN(t2), YARN, instancename, user)
- and slider-live-instances(YARN(t2))[0].rpcPort != 0
- and rpc-connection(slider-live-instances(YARN(t2))[0], SliderClusterProtocol)</code></pre>
+<div class="codehilite"><pre><span class="n">exists</span> <span class="n">t2</span> <span class="n">where</span><span class="o">:</span>
+ <span class="n">t2</span> <span class="o">></span> <span class="n">t1</span>
+ <span class="n">and</span> <span class="n">slider</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">live</span><span class="p">(</span><span class="n">YARN</span><span class="p">(</span><span class="n">t2</span><span class="p">),</span> <span class="n">YARN</span><span class="p">,</span> <span class="n">instancename</span><span class="p">,</span> <span class="n">user</span><span class="p">)</span>
+ <span class="n">and</span> <span class="n">slider</span><span class="o">-</span><span class="n">live</span><span class="o">-</span><span class="n">instances</span><span class="p">(</span><span class="n">YARN</span><span class="p">(</span><span class="n">t2</span><span class="p">))[</span><span class="mi">0</span><span class="p">].</span><span class="n">rpcPort</span> <span class="o">!=</span> <span class="mi">0</span>
+ <span class="n">and</span> <span class="n">rpc</span><span class="o">-</span><span class="n">connection</span><span class="p">(</span><span class="n">slider</span><span class="o">-</span><span class="n">live</span><span class="o">-</span><span class="n">instances</span><span class="p">(</span><span class="n">YARN</span><span class="p">(</span><span class="n">t2</span><span class="p">))[</span><span class="mi">0</span><span class="p">],</span> <span class="n">SliderClusterProtocol</span><span class="p">)</span>
+</pre></div>
<p>A test for accepting cluster requests is querying the cluster status
@@ -440,17 +466,19 @@ or during teardown.</p>
plus the number of requested instances , minus the number of instances for
which release requests have been made must match that of the desired number.</p>
<p>If the internal state of the Slider AM is defined as <code>AppState</code></p>
-<pre class="codehilite"><code>forall r in clusterspec.roles :
- r["yarn.component.instances"] ==
- AppState.Roles[r].live + AppState.Roles[r].requested - AppState.Roles[r].released</code></pre>
+<div class="codehilite"><pre><span class="n">forall</span> <span class="n">r</span> <span class="n">in</span> <span class="n">clusterspec</span><span class="p">.</span><span class="n">roles</span> <span class="o">:</span>
+ <span class="n">r</span><span class="p">[</span><span class="s">"yarn.component.instances"</span><span class="p">]</span> <span class="o">==</span>
+ <span class="n">AppState</span><span class="p">.</span><span class="n">Roles</span><span class="p">[</span><span class="n">r</span><span class="p">].</span><span class="n">live</span> <span class="o">+</span> <span class="n">AppState</span><span class="p">.</span><span class="n">Roles</span><span class="p">[</span><span class="n">r</span><span class="p">].</span><span class="n">requested</span> <span class="o">-</span> <span class="n">AppState</span><span class="p">.</span><span class="n">Roles</span><span class="p">[</span><span class="n">r</span><span class="p">].</span><span class="n">released</span>
+</pre></div>
<p>The <code>AppState</code> represents Slider's view of the external YARN system state, based on its
history of notifications received from YARN. </p>
<p>It is indirectly observable from the cluster state which an AM can be queried for</p>
-<pre class="codehilite"><code>forall r in AM.getJSONClusterStatus().roles :
- r["yarn.component.instances"] ==
- r["role.actual.instances"] + r["role.requested.instances"] - r["role.releasing.instances"]</code></pre>
+<div class="codehilite"><pre><span class="n">forall</span> <span class="n">r</span> <span class="n">in</span> <span class="n">AM</span><span class="p">.</span><span class="n">getJSONClusterStatus</span><span class="p">().</span><span class="n">roles</span> <span class="o">:</span>
+ <span class="n">r</span><span class="p">[</span><span class="s">"yarn.component.instances"</span><span class="p">]</span> <span class="o">==</span>
+ <span class="n">r</span><span class="p">[</span><span class="s">"role.actual.instances"</span><span class="p">]</span> <span class="o">+</span> <span class="n">r</span><span class="p">[</span><span class="s">"role.requested.instances"</span><span class="p">]</span> <span class="o">-</span> <span class="n">r</span><span class="p">[</span><span class="s">"role.releasing.instances"</span><span class="p">]</span>
+</pre></div>
<p>Slider does not consider it an error if the number of actual instances remains below
@@ -469,9 +497,10 @@ conclude that the role is somehow failin
entire application.</p>
<p>This has initially been implemented as a simple counter, with the cluster
option: <code>"slider.container.failure.threshold"</code> defining that threshold.</p>
-<pre class="codehilite"><code>let status = AM.getJSONClusterStatus()
-forall r in in status.roles :
- r["role.failed.instances"] < status.options["slider.container.failure.threshold"]</code></pre>
+<div class="codehilite"><pre><span class="n">let</span> <span class="n">status</span> <span class="o">=</span> <span class="n">AM</span><span class="p">.</span><span class="n">getJSONClusterStatus</span><span class="p">()</span>
+<span class="n">forall</span> <span class="n">r</span> <span class="n">in</span> <span class="n">in</span> <span class="n">status</span><span class="p">.</span><span class="n">roles</span> <span class="o">:</span>
+ <span class="n">r</span><span class="p">[</span><span class="s">"role.failed.instances"</span><span class="p">]</span> <span class="o"><</span> <span class="n">status</span><span class="p">.</span><span class="n">options</span><span class="p">[</span><span class="s">"slider.container.failure.threshold"</span><span class="p">]</span>
+</pre></div>
<h3 id="instance-startup-failure">Instance startup failure</h3>
@@ -497,7 +526,8 @@ to run/exiting on or nearly immediately.
<p>Create is simply <code>build</code> + <code>thaw</code> in sequence - the postconditions from the first
action are intended to match the preconditions of the second.</p>
<h2 id="action-freeze">Action: Freeze</h2>
-<pre class="codehilite"><code>freeze instancename [--wait time] [--message message]</code></pre>
+<div class="codehilite"><pre><span class="n">freeze</span> <span class="n">instancename</span> <span class="p">[</span><span class="o">--</span><span class="n">wait</span> <span class="n">time</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">message</span> <span class="n">message</span><span class="p">]</span>
+</pre></div>
<p>The <em>freeze</em> action "freezes" the cluster: all its nodes running in the YARN
@@ -506,10 +536,11 @@ cluster are stopped, leaving all the per
freeze is invoked on an already frozen cluster</p>
<h4 id="preconditions_4">Preconditions</h4>
<p>The cluster name is valid and it matches a known cluster </p>
-<pre class="codehilite"><code>if not valid-instance-name(instancename) : raise SliderException(EXIT_COMMAND_ARGUMENT_ERROR)
+<div class="codehilite"><pre><span class="k">if</span> <span class="n">not</span> <span class="n">valid</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">name</span><span class="p">(</span><span class="n">instancename</span><span class="p">)</span> <span class="o">:</span> <span class="n">raise</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_COMMAND_ARGUMENT_ERROR</span><span class="p">)</span>
-if not is-file(HDFS, application-instance-path(HDFS, instancename)) :
- raise SliderException(EXIT_UNKNOWN_INSTANCE)</code></pre>
+<span class="k">if</span> <span class="n">not</span> <span class="n">is</span><span class="o">-</span><span class="n">file</span><span class="p">(</span><span class="n">HDFS</span><span class="p">,</span> <span class="n">application</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">HDFS</span><span class="p">,</span> <span class="n">instancename</span><span class="p">))</span> <span class="o">:</span>
+ <span class="n">raise</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_UNKNOWN_INSTANCE</span><span class="p">)</span>
+</pre></div>
<h4 id="postconditions_4">Postconditions</h4>
@@ -519,13 +550,15 @@ until the cluster has finished or the wa
<p>If the <code>--message</code> argument specified a message -it must appear in the
YARN logs as the reason the cluster was frozen.</p>
<p>The outcome should be the same:</p>
-<pre class="codehilite"><code>not slider-instance-live(YARN', instancename)</code></pre>
+<div class="codehilite"><pre><span class="n">not</span> <span class="n">slider</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">live</span><span class="p">(</span><span class="n">YARN</span><span class="err">'</span><span class="p">,</span> <span class="n">instancename</span><span class="p">)</span>
+</pre></div>
<h2 id="action-flex">Action: Flex</h2>
<p>Flex the cluster size: add or remove roles. </p>
-<pre class="codehilite"><code>flex instancename
-components:List[(String, int)]</code></pre>
+<div class="codehilite"><pre><span class="n">flex</span> <span class="n">instancename</span>
+<span class="nl">components:</span><span class="n">List</span><span class="p">[(</span><span class="n">String</span><span class="p">,</span> <span class="kt">int</span><span class="p">)]</span>
+</pre></div>
<ol>
@@ -534,23 +567,26 @@ components:List[(String, int)]</code></p
which will change the desired steady-state of the application</li>
</ol>
<h4 id="preconditions_5">Preconditions</h4>
-<pre class="codehilite"><code>if not is-file(HDFS, cluster-json-path(HDFS, instancename)) :
- raise SliderException(EXIT_UNKNOWN_INSTANCE)</code></pre>
+<div class="codehilite"><pre><span class="k">if</span> <span class="n">not</span> <span class="n">is</span><span class="o">-</span><span class="n">file</span><span class="p">(</span><span class="n">HDFS</span><span class="p">,</span> <span class="n">cluster</span><span class="o">-</span><span class="n">json</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">HDFS</span><span class="p">,</span> <span class="n">instancename</span><span class="p">))</span> <span class="o">:</span>
+ <span class="n">raise</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_UNKNOWN_INSTANCE</span><span class="p">)</span>
+</pre></div>
<h4 id="postconditions_5">Postconditions</h4>
-<pre class="codehilite"><code>let originalSpec = data(HDFS, cluster-json-path(HDFS, instancename))
+<div class="codehilite"><pre><span class="n">let</span> <span class="n">originalSpec</span> <span class="o">=</span> <span class="n">data</span><span class="p">(</span><span class="n">HDFS</span><span class="p">,</span> <span class="n">cluster</span><span class="o">-</span><span class="n">json</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">HDFS</span><span class="p">,</span> <span class="n">instancename</span><span class="p">))</span>
-let updatedSpec = originalspec where:
- forall (name, size) in components :
- updatedSpec.roles[name]["yarn.component.instances"] == size
-data(HDFS', cluster-json-path(HDFS', instancename)) == updatedSpec
-rpc-connection(slider-live-instances(YARN(t2))[0], SliderClusterProtocol)
-let flexed = rpc-connection(slider-live-instances(YARN(t2))[0], SliderClusterProtocol).flexClusterupdatedSpec)</code></pre>
+<span class="n">let</span> <span class="n">updatedSpec</span> <span class="o">=</span> <span class="n">originalspec</span> <span class="n">where</span><span class="o">:</span>
+ <span class="n">forall</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">size</span><span class="p">)</span> <span class="n">in</span> <span class="n">components</span> <span class="o">:</span>
+ <span class="n">updatedSpec</span><span class="p">.</span><span class="n">roles</span><span class="p">[</span><span class="n">name</span><span class="p">][</span><span class="s">"yarn.component.instances"</span><span class="p">]</span> <span class="o">==</span> <span class="n">size</span>
+<span class="n">data</span><span class="p">(</span><span class="n">HDFS</span><span class="err">'</span><span class="p">,</span> <span class="n">cluster</span><span class="o">-</span><span class="n">json</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">HDFS</span><span class="err">'</span><span class="p">,</span> <span class="n">instancename</span><span class="p">))</span> <span class="o">==</span> <span class="n">updatedSpec</span>
+<span class="n">rpc</span><span class="o">-</span><span class="n">connection</span><span class="p">(</span><span class="n">slider</span><span class="o">-</span><span class="n">live</span><span class="o">-</span><span class="n">instances</span><span class="p">(</span><span class="n">YARN</span><span class="p">(</span><span class="n">t2</span><span class="p">))[</span><span class="mi">0</span><span class="p">],</span> <span class="n">SliderClusterProtocol</span><span class="p">)</span>
+<span class="n">let</span> <span class="n">flexed</span> <span class="o">=</span> <span class="n">rpc</span><span class="o">-</span><span class="n">connection</span><span class="p">(</span><span class="n">slider</span><span class="o">-</span><span class="n">live</span><span class="o">-</span><span class="n">instances</span><span class="p">(</span><span class="n">YARN</span><span class="p">(</span><span class="n">t2</span><span class="p">))[</span><span class="mi">0</span><span class="p">],</span> <span class="n">SliderClusterProtocol</span><span class="p">).</span><span class="n">flexClusterupdatedSpec</span><span class="p">)</span>
+</pre></div>
<h4 id="am-actions-on-flex">AM actions on flex</h4>
-<pre class="codehilite"><code>boolean SliderAppMaster.flexCluster(ClusterDescription updatedSpec)</code></pre>
+<div class="codehilite"><pre><span class="n">boolean</span> <span class="n">SliderAppMaster</span><span class="p">.</span><span class="n">flexCluster</span><span class="p">(</span><span class="n">ClusterDescription</span> <span class="n">updatedSpec</span><span class="p">)</span>
+</pre></div>
<p>If the cluster is in a state where flexing is possible (i.e. it is not in teardown),
@@ -558,13 +594,15 @@ then <code>AppState</code> is updated wi
return once all requests to add or remove role instances have been queued,
and be <code>True</code> iff the desired steady state of the cluster has been changed.</p>
<h4 id="preconditions_6">Preconditions</h4>
-<pre class="codehilite"><code> well-defined-application-instance(HDFS, updatedSpec)</code></pre>
+<div class="codehilite"><pre> <span class="n">well</span><span class="o">-</span><span class="n">defined</span><span class="o">-</span><span class="n">application</span><span class="o">-</span><span class="n">instance</span><span class="p">(</span><span class="n">HDFS</span><span class="p">,</span> <span class="n">updatedSpec</span><span class="p">)</span>
+</pre></div>
<h4 id="postconditions_6">Postconditions</h4>
-<pre class="codehilite"><code>forall role in AppState.Roles.keys:
- AppState'.Roles'[role].desiredCount = updatedSpec[roles]["yarn.component.instances"]
-result = AppState' != AppState</code></pre>
+<div class="codehilite"><pre><span class="n">forall</span> <span class="n">role</span> <span class="n">in</span> <span class="n">AppState</span><span class="p">.</span><span class="n">Roles</span><span class="p">.</span><span class="n">keys</span><span class="o">:</span>
+ <span class="n">AppState</span><span class="err">'</span><span class="p">.</span><span class="n">Roles</span><span class="err">'</span><span class="p">[</span><span class="n">role</span><span class="p">].</span><span class="n">desiredCount</span> <span class="o">=</span> <span class="n">updatedSpec</span><span class="p">[</span><span class="n">roles</span><span class="p">][</span><span class="s">"yarn.component.instances"</span><span class="p">]</span>
+<span class="n">result</span> <span class="o">=</span> <span class="n">AppState</span><span class="err">'</span> <span class="o">!=</span> <span class="n">AppState</span>
+</pre></div>
<p>The flexing may change the desired steady state of the cluster, in which
@@ -576,37 +614,44 @@ satisfied.</p>
cluster has already been destroyed/is unknown, but not if it is
actually running.</p>
<h4 id="preconditions_7">Preconditions</h4>
-<pre class="codehilite"><code>if not valid-instance-name(instancename) : raise SliderException(EXIT_COMMAND_ARGUMENT_ERROR)
+<div class="codehilite"><pre><span class="k">if</span> <span class="n">not</span> <span class="n">valid</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">name</span><span class="p">(</span><span class="n">instancename</span><span class="p">)</span> <span class="o">:</span> <span class="n">raise</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_COMMAND_ARGUMENT_ERROR</span><span class="p">)</span>
-if slider-instance-live(YARN, instancename) : raise SliderException(EXIT_CLUSTER_IN_USE)</code></pre>
+<span class="k">if</span> <span class="n">slider</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">live</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">instancename</span><span class="p">)</span> <span class="o">:</span> <span class="n">raise</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_CLUSTER_IN_USE</span><span class="p">)</span>
+</pre></div>
<h4 id="postconditions_7">Postconditions</h4>
<p>The cluster directory and all its children do not exist</p>
-<pre class="codehilite"><code>not is-dir(HDFS', application-instance-path(HDFS', instancename))</code></pre>
+<div class="codehilite"><pre><span class="n">not</span> <span class="n">is</span><span class="o">-</span><span class="n">dir</span><span class="p">(</span><span class="n">HDFS</span><span class="err">'</span><span class="p">,</span> <span class="n">application</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">HDFS</span><span class="err">'</span><span class="p">,</span> <span class="n">instancename</span><span class="p">))</span>
+</pre></div>
<h2 id="action-status">Action: Status</h2>
-<pre class="codehilite"><code>status instancename [--out outfile]
-2</code></pre>
+<div class="codehilite"><pre><span class="n">status</span> <span class="n">instancename</span> <span class="p">[</span><span class="o">--</span><span class="n">out</span> <span class="n">outfile</span><span class="p">]</span>
+<span class="mi">2</span>
+</pre></div>
<h4 id="preconditions_8">Preconditions</h4>
-<pre class="codehilite"><code>if not slider-instance-live(YARN, instancename) : raise SliderException(EXIT_UNKNOWN_INSTANCE)</code></pre>
+<div class="codehilite"><pre><span class="k">if</span> <span class="n">not</span> <span class="n">slider</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">live</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">instancename</span><span class="p">)</span> <span class="o">:</span> <span class="n">raise</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_UNKNOWN_INSTANCE</span><span class="p">)</span>
+</pre></div>
<h4 id="postconditions_8">Postconditions</h4>
<p>The status of the application has been successfully queried and printed out:</p>
-<pre class="codehilite"><code>let status = slider-live-instances(YARN).rpcPort.getJSONClusterStatus()</code></pre>
+<div class="codehilite"><pre><span class="n">let</span> <span class="n">status</span> <span class="o">=</span> <span class="n">slider</span><span class="o">-</span><span class="n">live</span><span class="o">-</span><span class="n">instances</span><span class="p">(</span><span class="n">YARN</span><span class="p">).</span><span class="n">rpcPort</span><span class="p">.</span><span class="n">getJSONClusterStatus</span><span class="p">()</span>
+</pre></div>
<p>if the <code>outfile</code> value is not defined then the status appears part of stdout</p>
-<pre class="codehilite"><code>status in STDOUT'</code></pre>
+<div class="codehilite"><pre><span class="n">status</span> <span class="n">in</span> <span class="n">STDOUT</span><span class="err">'</span>
+</pre></div>
<p>otherwise, the outfile exists in the local filesystem</p>
-<pre class="codehilite"><code>(outfile != "") ==> data(LocalFS', outfile) == body
-(outfile != "") ==> body in STDOUT'</code></pre>
+<div class="codehilite"><pre><span class="p">(</span><span class="n">outfile</span> <span class="o">!=</span> <span class="s">""</span><span class="p">)</span> <span class="o">==></span> <span class="n">data</span><span class="p">(</span><span class="n">LocalFS</span><span class="err">'</span><span class="p">,</span> <span class="n">outfile</span><span class="p">)</span> <span class="o">==</span> <span class="n">body</span>
+<span class="p">(</span><span class="n">outfile</span> <span class="o">!=</span> <span class="s">""</span><span class="p">)</span> <span class="o">==></span> <span class="n">body</span> <span class="n">in</span> <span class="n">STDOUT</span><span class="err">'</span>
+</pre></div>
<h2 id="action-exists">Action: Exists</h2>
@@ -615,67 +660,75 @@ state.</p>
<p>In the running state; it is essentially the status
operation with only the exit code returned</p>
<h4 id="preconditions_9">Preconditions</h4>
-<pre class="codehilite"><code>if not is-file(HDFS, application-instance-path(HDFS, instancename)) :
- raise SliderException(EXIT_UNKNOWN_INSTANCE)</code></pre>
+<div class="codehilite"><pre><span class="k">if</span> <span class="n">not</span> <span class="n">is</span><span class="o">-</span><span class="n">file</span><span class="p">(</span><span class="n">HDFS</span><span class="p">,</span> <span class="n">application</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">HDFS</span><span class="p">,</span> <span class="n">instancename</span><span class="p">))</span> <span class="o">:</span>
+ <span class="n">raise</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_UNKNOWN_INSTANCE</span><span class="p">)</span>
+</pre></div>
<h4 id="postconditions_9">Postconditions</h4>
<p>The operation succeeds if the cluster is running and the RPC call returns the cluster
status.</p>
-<pre class="codehilite"><code>if live and not slider-instance-live(YARN, instancename):
- retcode = -1
-else:
- retcode = 0</code></pre>
+<div class="codehilite"><pre><span class="k">if</span> <span class="n">live</span> <span class="n">and</span> <span class="n">not</span> <span class="n">slider</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">live</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">instancename</span><span class="p">)</span><span class="o">:</span>
+ <span class="n">retcode</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span>
+<span class="nl">else:</span>
+ <span class="n">retcode</span> <span class="o">=</span> <span class="mi">0</span>
+</pre></div>
<h2 id="action-getconf">Action: getConf</h2>
<p>This returns the live client configuration of the cluster -the
site-xml file.</p>
-<pre class="codehilite"><code>getconf --format (xml|properties) --out [outfile]</code></pre>
+<div class="codehilite"><pre><span class="n">getconf</span> <span class="o">--</span><span class="n">format</span> <span class="p">(</span><span class="n">xml</span><span class="o">|</span><span class="n">properties</span><span class="p">)</span> <span class="o">--</span><span class="n">out</span> <span class="p">[</span><span class="n">outfile</span><span class="p">]</span>
+</pre></div>
<p><em>We may want to think hard about whether this is needed</em></p>
<h4 id="preconditions_10">Preconditions</h4>
-<pre class="codehilite"><code>if not slider-instance-live(YARN, instancename) : raise SliderException(EXIT_UNKNOWN_INSTANCE)</code></pre>
+<div class="codehilite"><pre><span class="k">if</span> <span class="n">not</span> <span class="n">slider</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">live</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">instancename</span><span class="p">)</span> <span class="o">:</span> <span class="n">raise</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_UNKNOWN_INSTANCE</span><span class="p">)</span>
+</pre></div>
<h4 id="postconditions_10">Postconditions</h4>
<p>The operation succeeds if the cluster status can be retrieved and saved to
the named file/printed to stdout in the format chosen</p>
-<pre class="codehilite"><code>let status = slider-live-instances(YARN).rpcPort.getJSONClusterStatus()
-let conf = status.clientProperties
-if format == "xml" :
- let body = status.clientProperties.asXmlDocument()
-else:
- let body = status.clientProperties.asProperties()
-
-if outfile != "" :
- data(LocalFS', outfile) == body
-else
- body in STDOUT'</code></pre>
+<div class="codehilite"><pre><span class="n">let</span> <span class="n">status</span> <span class="o">=</span> <span class="n">slider</span><span class="o">-</span><span class="n">live</span><span class="o">-</span><span class="n">instances</span><span class="p">(</span><span class="n">YARN</span><span class="p">).</span><span class="n">rpcPort</span><span class="p">.</span><span class="n">getJSONClusterStatus</span><span class="p">()</span>
+<span class="n">let</span> <span class="n">conf</span> <span class="o">=</span> <span class="n">status</span><span class="p">.</span><span class="n">clientProperties</span>
+<span class="k">if</span> <span class="n">format</span> <span class="o">==</span> <span class="s">"xml"</span> <span class="o">:</span>
+ <span class="n">let</span> <span class="n">body</span> <span class="o">=</span> <span class="n">status</span><span class="p">.</span><span class="n">clientProperties</span><span class="p">.</span><span class="n">asXmlDocument</span><span class="p">()</span>
+<span class="nl">else:</span>
+ <span class="n">let</span> <span class="n">body</span> <span class="o">=</span> <span class="n">status</span><span class="p">.</span><span class="n">clientProperties</span><span class="p">.</span><span class="n">asProperties</span><span class="p">()</span>
+
+<span class="k">if</span> <span class="n">outfile</span> <span class="o">!=</span> <span class="s">""</span> <span class="o">:</span>
+ <span class="n">data</span><span class="p">(</span><span class="n">LocalFS</span><span class="err">'</span><span class="p">,</span> <span class="n">outfile</span><span class="p">)</span> <span class="o">==</span> <span class="n">body</span>
+<span class="k">else</span>
+ <span class="n">body</span> <span class="n">in</span> <span class="n">STDOUT</span><span class="err">'</span>
+</pre></div>
<h2 id="action-list">Action: list</h2>
-<pre class="codehilite"><code>list [instancename]</code></pre>
+<div class="codehilite"><pre><span class="n">list</span> <span class="p">[</span><span class="n">instancename</span><span class="p">]</span>
+</pre></div>
<p>Lists all clusters of a user, or only the one given</p>
<h4 id="preconditions_11">Preconditions</h4>
<p>If a instancename is specified it must be in YARNs list of active or completed applications
of that user:</p>
-<pre class="codehilite"><code>if instancename != "" and [] == yarn-application-instances(YARN, instancename, user)
- raise SliderException(EXIT_UNKNOWN_INSTANCE)</code></pre>
+<div class="codehilite"><pre><span class="k">if</span> <span class="n">instancename</span> <span class="o">!=</span> <span class="s">""</span> <span class="n">and</span> <span class="p">[]</span> <span class="o">==</span> <span class="n">yarn</span><span class="o">-</span><span class="n">application</span><span class="o">-</span><span class="n">instances</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">instancename</span><span class="p">,</span> <span class="n">user</span><span class="p">)</span>
+ <span class="n">raise</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_UNKNOWN_INSTANCE</span><span class="p">)</span>
+</pre></div>
<h4 id="postconditions_11">Postconditions</h4>
<p>If no instancename was given, all slider applications of that user are listed,
else only the one running (or one of the finished ones)</p>
-<pre class="codehilite"><code>if instancename == "" :
- forall a in yarn-application-instances(YARN, user) :
- a.toString() in STDOUT'
-else
- let e = yarn-application-instances(YARN, instancename, user)
- e.toString() in STDOUT'</code></pre>
+<div class="codehilite"><pre><span class="k">if</span> <span class="n">instancename</span> <span class="o">==</span> <span class="s">""</span> <span class="o">:</span>
+ <span class="n">forall</span> <span class="n">a</span> <span class="n">in</span> <span class="n">yarn</span><span class="o">-</span><span class="n">application</span><span class="o">-</span><span class="n">instances</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">user</span><span class="p">)</span> <span class="o">:</span>
+ <span class="n">a</span><span class="p">.</span><span class="n">toString</span><span class="p">()</span> <span class="n">in</span> <span class="n">STDOUT</span><span class="err">'</span>
+<span class="k">else</span>
+ <span class="n">let</span> <span class="n">e</span> <span class="o">=</span> <span class="n">yarn</span><span class="o">-</span><span class="n">application</span><span class="o">-</span><span class="n">instances</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">instancename</span><span class="p">,</span> <span class="n">user</span><span class="p">)</span>
+ <span class="n">e</span><span class="p">.</span><span class="n">toString</span><span class="p">()</span> <span class="n">in</span> <span class="n">STDOUT</span><span class="err">'</span>
+</pre></div>
<h2 id="action-killcontainer">Action: killcontainer</h2>
@@ -683,38 +736,44 @@ else
<em>without flexing the cluster size</em>. As a result, the cluster will detect the
failure and attempt to recover from the failure by instantiating a new instance
of the cluster</p>
-<pre class="codehilite"><code>killcontainer cluster --id container-id</code></pre>
+<div class="codehilite"><pre><span class="n">killcontainer</span> <span class="n">cluster</span> <span class="o">--</span><span class="n">id</span> <span class="n">container</span><span class="o">-</span><span class="n">id</span>
+</pre></div>
<h4 id="preconditions_12">Preconditions</h4>
-<pre class="codehilite"><code>if not slider-instance-live(YARN, instancename) : raise SliderException(EXIT_UNKNOWN_INSTANCE)
+<div class="codehilite"><pre><span class="k">if</span> <span class="n">not</span> <span class="n">slider</span><span class="o">-</span><span class="n">instance</span><span class="o">-</span><span class="n">live</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">instancename</span><span class="p">)</span> <span class="o">:</span> <span class="n">raise</span> <span class="n">SliderException</span><span class="p">(</span><span class="n">EXIT_UNKNOWN_INSTANCE</span><span class="p">)</span>
-exists c in slider-app-containers(YARN, instancename, user) where c.id == container-id
+<span class="n">exists</span> <span class="n">c</span> <span class="n">in</span> <span class="n">slider</span><span class="o">-</span><span class="n">app</span><span class="o">-</span><span class="n">containers</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">instancename</span><span class="p">,</span> <span class="n">user</span><span class="p">)</span> <span class="n">where</span> <span class="n">c</span><span class="p">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">container</span><span class="o">-</span><span class="n">id</span>
-let status := AM.getJSONClusterStatus()
-exists role = status.instances where container-id in status.instances[role].values</code></pre>
+<span class="n">let</span> <span class="n">status</span> <span class="o">:=</span> <span class="n">AM</span><span class="p">.</span><span class="n">getJSONClusterStatus</span><span class="p">()</span>
+<span class="n">exists</span> <span class="n">role</span> <span class="o">=</span> <span class="n">status</span><span class="p">.</span><span class="n">instances</span> <span class="n">where</span> <span class="n">container</span><span class="o">-</span><span class="n">id</span> <span class="n">in</span> <span class="n">status</span><span class="p">.</span><span class="n">instances</span><span class="p">[</span><span class="n">role</span><span class="p">].</span><span class="n">values</span>
+</pre></div>
<h4 id="postconditions_12">Postconditions</h4>
<p>The container is not in the list of containers in the cluster</p>
-<pre class="codehilite"><code>not exists c in containers(YARN) where c.id == container-id</code></pre>
+<div class="codehilite"><pre><span class="n">not</span> <span class="n">exists</span> <span class="n">c</span> <span class="n">in</span> <span class="n">containers</span><span class="p">(</span><span class="n">YARN</span><span class="p">)</span> <span class="n">where</span> <span class="n">c</span><span class="p">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">container</span><span class="o">-</span><span class="n">id</span>
+</pre></div>
<p>And implicitly, not in the running containers of that application</p>
-<pre class="codehilite"><code>not exists c in slider-app-containers(YARN', instancename, user) where c.id == container-id</code></pre>
+<div class="codehilite"><pre><span class="n">not</span> <span class="n">exists</span> <span class="n">c</span> <span class="n">in</span> <span class="n">slider</span><span class="o">-</span><span class="n">app</span><span class="o">-</span><span class="n">containers</span><span class="p">(</span><span class="n">YARN</span><span class="err">'</span><span class="p">,</span> <span class="n">instancename</span><span class="p">,</span> <span class="n">user</span><span class="p">)</span> <span class="n">where</span> <span class="n">c</span><span class="p">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">container</span><span class="o">-</span><span class="n">id</span>
+</pre></div>
<p>At some time <code>t1 > t</code>, the status of the application (<code>AM'</code>) will be updated to reflect
that YARN has notified the AM of the loss of the container</p>
-<pre class="codehilite"><code>let status' = AM'.getJSONClusterStatus()
-len(status'.instances[role]) < len(status.instances[role])
-status'.roles[role]["role.failed.instances"] == status'.roles[role]["role.failed.instances"]+1</code></pre>
+<div class="codehilite"><pre><span class="n">let</span> <span class="n">status</span><span class="err">'</span> <span class="o">=</span> <span class="n">AM</span><span class="err">'</span><span class="p">.</span><span class="n">getJSONClusterStatus</span><span class="p">()</span>
+<span class="n">len</span><span class="p">(</span><span class="n">status</span><span class="err">'</span><span class="p">.</span><span class="n">instances</span><span class="p">[</span><span class="n">role</span><span class="p">])</span> <span class="o"><</span> <span class="n">len</span><span class="p">(</span><span class="n">status</span><span class="p">.</span><span class="n">instances</span><span class="p">[</span><span class="n">role</span><span class="p">])</span>
+<span class="n">status</span><span class="err">'</span><span class="p">.</span><span class="n">roles</span><span class="p">[</span><span class="n">role</span><span class="p">][</span><span class="s">"role.failed.instances"</span><span class="p">]</span> <span class="o">==</span> <span class="n">status</span><span class="err">'</span><span class="p">.</span><span class="n">roles</span><span class="p">[</span><span class="n">role</span><span class="p">][</span><span class="s">"role.failed.instances"</span><span class="p">]</span><span class="o">+</span><span class="mi">1</span>
+</pre></div>
<p>At some time <code>t2 > t1</code> in the future, the size of the containers of the application
in the YARN cluster <code>YARN''</code> will be as before </p>
-<pre class="codehilite"><code>let status'' = AM''.getJSONClusterStatus()
-len(status''.instances[r] == len(status.instances[r])</code></pre>
+<div class="codehilite"><pre><span class="n">let</span> <span class="n">status</span><span class="err">''</span> <span class="o">=</span> <span class="n">AM</span><span class="err">''</span><span class="p">.</span><span class="n">getJSONClusterStatus</span><span class="p">()</span>
+<span class="n">len</span><span class="p">(</span><span class="n">status</span><span class="err">''</span><span class="p">.</span><span class="n">instances</span><span class="p">[</span><span class="n">r</span><span class="p">]</span> <span class="o">==</span> <span class="n">len</span><span class="p">(</span><span class="n">status</span><span class="p">.</span><span class="n">instances</span><span class="p">[</span><span class="n">r</span><span class="p">])</span>
+</pre></div>
</div>
<div id="footer">