You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@slider.apache.org by bu...@apache.org on 2014/10/09 02:28:02 UTC
svn commit: r925106 - in /websites/staging/slider/trunk/content: ./
docs/manpage.html docs/slider_specs/resource_specification.html
Author: buildbot
Date: Thu Oct 9 00:28:02 2014
New Revision: 925106
Log:
Staging update by buildbot for slider
Modified:
websites/staging/slider/trunk/content/ (props changed)
websites/staging/slider/trunk/content/docs/manpage.html
websites/staging/slider/trunk/content/docs/slider_specs/resource_specification.html
Propchange: websites/staging/slider/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Thu Oct 9 00:28:02 2014
@@ -1 +1 @@
-1629803
+1630262
Modified: websites/staging/slider/trunk/content/docs/manpage.html
==============================================================================
--- websites/staging/slider/trunk/content/docs/manpage.html (original)
+++ websites/staging/slider/trunk/content/docs/manpage.html Thu Oct 9 00:28:02 2014
@@ -178,14 +178,14 @@ Latest release: <strong>0.40-incubating<
<h2 id="synopsis">SYNOPSIS</h2>
<p>Slider enables applications to be dynamically created on a YARN-managed datacenter.
The program can be used to create, pause, and shutdown the application. It can also be used to list current active
-and existing but not running "frozen" application instances.</p>
+and existing but not running "stopped" application instances.</p>
<h2 id="concepts">CONCEPTS</h2>
<ol>
<li>
<p>A <em>Slider application</em> is an application packaged to be deployed by Slider. It consists of one or more distributed <em>components</em> </p>
</li>
<li>
-<p>A <em>Slider application instance</em> is a slider application configured to be deployable on a specific YARN cluster, with a specific configuration. An instance can be <em>live</em> -actually running- or <em>frozen</em>. When frozen all its configuration details and instance-specific data are preserved on HDFS.</p>
+<p>A <em>Slider application instance</em> is a slider application configured to be deployable on a specific YARN cluster, with a specific configuration. An instance can be <em>live</em> -actually running- or <em>stopped</em>. When stopped all its configuration details and instance-specific data are preserved on HDFS.</p>
</li>
<li>
<p>An <em>image</em> is a <em>tar.gz</em> file containing binaries used to create the application. 1. Images are kept in the HDFS filesystem and identified by their path names; filesystem permissions can be used to share images amongst users.</p>
@@ -212,13 +212,13 @@ and existing but not running "frozen" ap
<p>A user can create an application instance.</p>
</li>
<li>
-<p>A live instances can be <em>frozen</em>, saving its final state to its application instance state directory. All running components are shut down.</p>
+<p>A live instances can be <em>stopped</em>, saving its final state to its application instance state directory. All running components are shut down.</p>
</li>
<li>
-<p>A frozen instance can be <em>thawed</em> -a its components started on or near the servers where they were previously running.</p>
+<p>A stopped instance can be <em>started</em> -a its components started on or near the servers where they were previously running.</p>
</li>
<li>
-<p>A frozen instance can be <em>destroyed</em>. </p>
+<p>A stopped instance can be <em>destroyed</em>. </p>
</li>
<li>
<p>Running instances can be listed. </p>
@@ -367,6 +367,24 @@ allocations, which must use the <code>--
<p>If unset, the zookeeper quorum defined in the property <code>slider.zookeeper.quorum</code>
is used</p>
+<h5 id="-queue-queue-name"><code>--queue <queue name></code></h5>
+<p>The queue to deploy the application to. By default, YARN will pick the queue.</p>
+<p>Example</p>
+<div class="codehilite"><pre><span class="o">--</span><span class="n">queue</span> <span class="n">applications</span>
+</pre></div>
+
+
+<h4 id="examples"><code>Examples</code></h4>
+<p>Create an application by providing <code>template</code> and <code>resources</code>.</p>
+<div class="codehilite"><pre><span class="n">create</span> <span class="n">hbase1</span> <span class="o">--</span><span class="n">template</span> <span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">work</span><span class="o">/</span><span class="n">hbase</span><span class="o">/</span><span class="n">appConfig</span><span class="p">.</span><span class="n">json</span> <span class="o">--</span><span class="n">resources</span> <span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">work</span><span class="o">/</span><span class="n">hbase</span><span class="o">/</span><span class="n">resources</span><span class="p">.</span><span class="n">json</span>
+</pre></div>
+
+
+<p>Create an application by providing <code>template</code> and <code>resources</code> and <code>queue</code>.</p>
+<div class="codehilite"><pre><span class="n">create</span> <span class="n">hbase1</span> <span class="o">--</span><span class="n">template</span> <span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">work</span><span class="o">/</span><span class="n">hbase</span><span class="o">/</span><span class="n">appConfig</span><span class="p">.</span><span class="n">json</span> <span class="o">--</span><span class="n">resources</span> <span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">work</span><span class="o">/</span><span class="n">hbase</span><span class="o">/</span><span class="n">resources</span><span class="p">.</span><span class="n">json</span> <span class="o">--</span><span class="n">queue</span> <span class="n">default</span>
+</pre></div>
+
+
<h3 id="destroy-name"><code>destroy <name></code></h3>
<p>Destroy a (stopped) applicaton instance .</p>
<p>Important: This deletes all persistent data</p>
@@ -425,17 +443,17 @@ cluster</p>
</pre></div>
-<h3 id="freeze-name-force-wait-time-message-text"><code>freeze <name> [--force] [--wait time] [--message text]</code></h3>
-<p>(<strong>freeze</strong> has been renamed to <strong>stop</strong> in develop branch)</p>
-<p>freeze the application instance. The running application is stopped. Its settings are retained in HDFS.</p>
-<p>The <code>--wait</code> argument can specify a time in seconds to wait for the application instance to be frozen.</p>
-<p>The <code>--force</code> flag causes YARN asked directly to terminate the application instance.
-The <code>--message</code> argument supplies an optional text message to be used in
-the request: this will appear in the application's diagnostics in the YARN RM UI.</p>
-<p>If an unknown (or already frozen) application instance is named, no error is returned.</p>
-<p>Examples</p>
-<div class="codehilite"><pre><span class="n">slider</span> <span class="n">freeze</span> <span class="n">instance1</span> <span class="o">--</span><span class="n">wait</span> 30
-<span class="n">slider</span> <span class="n">freeze</span> <span class="n">instance2</span> <span class="o">--</span><span class="n">force</span> <span class="o">--</span><span class="n">message</span> "<span class="n">maintenance</span> <span class="n">session</span>"
+<h3 id="install-package-name-name-of-the-package-package-package-file-replacepkg"><code>install-package --name <name of the package> --package <package file> [--replacepkg]</code></h3>
+<p>Install the application package to the default package location for the user under ~/.slider/package/<name>. This is the location referred to by the appConfig.json file provided in the --template parameter in the create command.</p>
+<h5 id="-name-name-of-the-package"><code>--name <name of the package></code></h5>
+<p>Name of the package. It may be the same as the name provided in the metainfo.xml. Ensure that the same value is used in the default application package location specified in the default appConfig.json file.</p>
+<h5 id="-package-package-file"><code>--package <package file></code></h5>
+<p>Location of the package on local disk.</p>
+<h5 id="-replacepkg"><code>--replacepkg</code></h5>
+<p>Optional. Whether to overwrite an already installed package.</p>
+<p>Example</p>
+<div class="codehilite"><pre><span class="n">slider</span> <span class="n">install</span><span class="o">-</span><span class="n">package</span> <span class="o">--</span><span class="n">name</span> <span class="n">HBASE</span> <span class="o">--</span><span class="n">package</span> <span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">work</span><span class="o">/</span><span class="n">package</span><span class="o">/</span><span class="n">hbase</span><span class="o">/</span><span class="n">slider</span><span class="o">-</span><span class="n">hbase</span><span class="o">-</span><span class="n">app</span><span class="o">-</span><span class="n">package</span><span class="o">-</span>0<span class="p">.</span>98<span class="p">.</span>4<span class="o">-</span><span class="n">hadoop2</span><span class="p">.</span><span class="n">zip</span>
+<span class="n">slider</span> <span class="n">install</span><span class="o">-</span><span class="n">package</span> <span class="o">--</span><span class="n">name</span> <span class="n">HBASE</span> <span class="o">--</span><span class="n">package</span> <span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">work</span><span class="o">/</span><span class="n">package</span><span class="o">/</span><span class="n">hbase</span><span class="o">/</span><span class="n">slider</span><span class="o">-</span><span class="n">hbase</span><span class="o">-</span><span class="n">app</span><span class="o">-</span><span class="n">package</span><span class="o">-</span>0<span class="p">.</span>98<span class="p">.</span>4<span class="o">-</span><span class="n">hadoop2</span><span class="p">.</span><span class="n">zip</span> <span class="o">--</span><span class="n">replacepkg</span>
</pre></div>
@@ -490,10 +508,20 @@ is not found, the command returns the ex
<p>List all services of the service type and optionally the name. </p>
<h4 id="slider-registry-listconf-name-name-internal-servicetype-servicetype"><code>slider registry --listconf [--name <name>] [--internal] [--servicetype <servicetype>]</code></h4>
<p>List the configurations exported by of a named application</p>
-<h4 id="slider-registry-getconf-configuration-format-xmljsonproperties-dest-path-internal-get-the-configuration"><code>slider registry --getconf <configuration> [--format (xml|json|properties)] [--dest <path>] [--internal]</code> get the configuration</h4>
+<h4 id="slider-registry-getconf-configuration-format-xmljsonproperties-dest-path-internal"><code>slider registry --getconf <configuration> [--format (xml|json|properties)] [--dest <path>] [--internal]</code></h4>
<p>Get a named configuration in a chosen format. Default: XML</p>
<p><code>--dest <path></code> : the filename or directory to save a configuration to.
<code>--format (xml|json|properties)</code> defines the output format</p>
+<h3 id="start-name-wait-time"><code>start <name> [--wait time</code>]</h3>
+<p>(<strong>start</strong> used to be <strong>thaw</strong> in develop branch)</p>
+<p>Resume a stopped application instance, recreating it from its previously saved state. This will include a best-effort attempt to create the same number of nodes as before, though their locations may be different.</p>
+<p>Examples:</p>
+<div class="codehilite"><pre><span class="n">slider</span> <span class="n">start</span> <span class="n">instance2</span>
+<span class="n">slider</span> <span class="n">start</span> <span class="n">instance1</span> <span class="o">--</span><span class="n">wait</span> 60
+</pre></div>
+
+
+<p>If the application instance is already running, this command will not affect it.</p>
<h3 id="status-name-out-filename"><code>status <name> [--out <filename>]</code></h3>
<p>Get the status of the named application instance in JSON format. A filename can be used to
specify the destination.</p>
@@ -504,16 +532,20 @@ specify the destination.</p>
</pre></div>
-<h3 id="thaw-name-wait-time"><code>thaw <name> [--wait time</code>]</h3>
-<p>(<strong>thaw</strong> has been renamed to <strong>start</strong> in develop branch)</p>
-<p>Resume a frozen application instance, recreating it from its previously saved state. This will include a best-effort attempt to create the same number of nodes as before, though their locations may be different.</p>
-<p>Examples:</p>
-<div class="codehilite"><pre><span class="n">slider</span> <span class="n">thaw</span> <span class="n">instance2</span>
-<span class="n">slider</span> <span class="n">thaw</span> <span class="n">instance1</span> <span class="o">--</span><span class="n">wait</span> 60
+<h3 id="stop-name-force-wait-time-message-text"><code>stop <name> [--force] [--wait time] [--message text]</code></h3>
+<p>(<strong>stop</strong> used to be <strong>freeze</strong> in earlier releases)</p>
+<p>stop the application instance. The running application is stopped. Its settings are retained in HDFS.</p>
+<p>The <code>--wait</code> argument can specify a time in seconds to wait for the application instance to be stopped.</p>
+<p>The <code>--force</code> flag causes YARN asked directly to terminate the application instance.
+The <code>--message</code> argument supplies an optional text message to be used in
+the request: this will appear in the application's diagnostics in the YARN RM UI.</p>
+<p>If an unknown (or already stopped) application instance is named, no error is returned.</p>
+<p>Examples</p>
+<div class="codehilite"><pre><span class="n">slider</span> <span class="n">stop</span> <span class="n">instance1</span> <span class="o">--</span><span class="n">wait</span> 30
+<span class="n">slider</span> <span class="n">stop</span> <span class="n">instance2</span> <span class="o">--</span><span class="n">force</span> <span class="o">--</span><span class="n">message</span> "<span class="n">maintenance</span> <span class="n">session</span>"
</pre></div>
-<p>If the application instance is already running, this command will not affect it.</p>
<h3 id="version"><code>version</code></h3>
<p>The command <code>slider version</code> prints out information about the compiled
Slider application, the version of Hadoop against which it was built -and
Modified: websites/staging/slider/trunk/content/docs/slider_specs/resource_specification.html
==============================================================================
--- websites/staging/slider/trunk/content/docs/slider_specs/resource_specification.html (original)
+++ websites/staging/slider/trunk/content/docs/slider_specs/resource_specification.html Thu Oct 9 00:28:02 2014
@@ -173,8 +173,13 @@ Latest release: <strong>0.40-incubating<
-->
<h1 id="apache-slider-resource-specification">Apache Slider Resource Specification</h1>
+<ul>
+<li><a href="#failurepolicy">Container Failure Policy</a></li>
+<li><a href="#labels">Using Labels</a></li>
+<li><a href="#logagg">Specifying Log Aggregation</a></li>
+</ul>
<p>Resource specification is an input to Slider to specify the Yarn resource needs for each component type that belong to the application.</p>
-<p>An example resource requirement for an application that has two components "master" and "worker" is as follows. Slider will automatically add the requirements for the AppMaster for the application. This compoent is named "slider-appmaster".</p>
+<p>An example resource requirement for an application that has two components "master" and "worker" is as follows. Slider will automatically add the requirements for the AppMaster for the application. This component is named "slider-appmaster".</p>
<p>Some parameters that can be specified for a component instance include:</p>
<ul>
<li><code>yarn.memory</code>: amount of memory requried for the component instance</li>
@@ -209,7 +214,7 @@ Latest release: <strong>0.40-incubating<
</pre></div>
-<h2 id="container-failure-policy">Container Failure Policy</h2>
+<h2 id="wzxhzdk5wzxhzdk6container-failure-policy"><a name="failurepolicy"></a>Container Failure Policy</h2>
<p>YARN containers hosting component instances may fail. This can happen because of</p>
<ol>
<li>A problem in the configuration of the instance.</li>
@@ -304,7 +309,7 @@ with the exit code 73, <code>EXIT_DEPLOY
are requested, the failure threshold per hour is double that of the number of masters.</p>
<p>There are ten worker components requested; the failure threshold for these
components is overridden to be fifteen. This allows all workers to fail and
-the cluster to recover âbut only anothe five failures would be tolerated
+the cluster to recover âbut only another five failures would be tolerated
for the remaining hour.</p>
<p>These failure thresholds are all heuristics. When initially configuring an
application instance, low thresholds reduce the disruption caused by components
@@ -312,6 +317,51 @@ which are frequently failing due to conf
<p>In a production application, large failure thresholds and/or shorter windows
ensures that the application is resilient to transient failures of the underlying
YARN cluster and hardware.</p>
+<h2 id="wzxhzdk7wzxhzdk8using-labels"><a name="labels"></a>Using Labels</h2>
+<p>The resources.json file can be used to specify the labels to be used when allocating containers for the components. The details of the YARN Label feature can be found at <a href="https://issues.apache.org/jira/browse/YARN-796">YARN-796</a>.</p>
+<p>In summary:
+<em> Nodes can be assigned one or more labels
+</em> Capacity Queues can be defined with access to one or more labels
+<em> Ensure application components are associated with appropriate label expressions
+</em> Create the application using specific queue</p>
+<p>This way, you can gurantee that a certain set of nodes are reserved for an application or for a component within an application.</p>
+<p>Label expression is specified through property "yarn.label.expression". When no label expression is specified then it is assummed that only non-labeled nodes are used when allocating containers for component instances.</p>
+<p>If label expression is specified for slider-appmaster then it also becomes the default label expression for all component. To take advantage of default label expression leave out the property (see HBASE_REGIONSERVER in the example). Label expression with empty string ("yarn.label.expression":"") means nodes without labels. </p>
+<h3 id="example_1">Example</h3>
+<p>Here is a <code>resource.json</code> file for an HBase cluster which uses labels. The label for the application instance is "hbase1" and the label expression for the HBASE_MASTER components is "hbase1_master". HBASE_REGIONSERVER instances will automatically use label "hbase1". Alternatively, if you specify ("yarn.label.expression":"") for HBASE_REGIONSERVER then the containers will only be allocated on nodes with no labels.</p>
+<div class="codehilite"><pre><span class="p">{</span>
+ "<span class="n">schema</span>"<span class="p">:</span> "<span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">example</span><span class="p">.</span><span class="n">org</span><span class="o">/</span><span class="n">specification</span><span class="o">/</span><span class="n">v2</span><span class="p">.</span>0<span class="p">.</span>0"<span class="p">,</span>
+ "<span class="n">metadata</span>"<span class="p">:</span> <span class="p">{</span>
+ <span class="p">},</span>
+ "<span class="k">global</span>"<span class="p">:</span> <span class="p">{</span>
+ <span class="p">},</span>
+ "<span class="n">components</span>"<span class="p">:</span> <span class="p">{</span>
+ "<span class="n">HBASE_MASTER</span>"<span class="p">:</span> <span class="p">{</span>
+ "<span class="n">yarn</span><span class="p">.</span><span class="n">role</span><span class="p">.</span><span class="n">priority</span>"<span class="p">:</span> "1"<span class="p">,</span>
+ "<span class="n">yarn</span><span class="p">.</span><span class="n">component</span><span class="p">.</span><span class="n">instances</span>"<span class="p">:</span> "1"<span class="p">,</span>
+ "<span class="n">yarn</span><span class="p">.</span><span class="n">label</span><span class="p">.</span><span class="n">expression</span>"<span class="p">:</span>"<span class="n">hbase1_master</span>"
+ <span class="p">},</span>
+ "<span class="n">HBASE_REGIONSERVER</span>"<span class="p">:</span> <span class="p">{</span>
+ "<span class="n">yarn</span><span class="p">.</span><span class="n">role</span><span class="p">.</span><span class="n">priority</span>"<span class="p">:</span> "1"<span class="p">,</span>
+ "<span class="n">yarn</span><span class="p">.</span><span class="n">component</span><span class="p">.</span><span class="n">instances</span>"<span class="p">:</span> "1"<span class="p">,</span>
+ <span class="p">},</span>
+ "<span class="n">slider</span><span class="o">-</span><span class="n">appmaster</span>"<span class="p">:</span> <span class="p">{</span>
+ "<span class="n">yarn</span><span class="p">.</span><span class="n">label</span><span class="p">.</span><span class="n">expression</span>"<span class="p">:</span>"<span class="n">hbase1</span>"
+ <span class="p">}</span>
+ <span class="p">}</span>
+<span class="p">}</span>
+</pre></div>
+
+
+<p>Specifically, for the above example you will need:
+<em> Create two labels, <code>hbase1</code> and <code>hbase1_master</code> (use yarn rmadmin commands)
+</em> Assign the labels to nodes (use yarn rmadmin commands)
+<em> Perform refresh queue (yarn -refreshqueue)
+</em> Create a queue by defining it in the capacity scheduler config
+<em> Allow the queue to access to the labels and ensure that appropriate min/max capacity is assigned
+</em> Perform refresh queue (yarn -refreshqueue)
+* Create the Slider application against the above queue using parameter <code>--queue</code> while creating the application</p>
+<h2 id="wzxhzdk9wzxhzdk10using-log-aggregation"><a name="logagg"></a>Using Log Aggregation</h2>
</div>
<div id="footer">