You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@slider.apache.org by bu...@apache.org on 2015/10/15 04:03:35 UTC
svn commit: r968967 - in /websites/staging/slider/trunk/content: ./
docs/configuration/resources.html
Author: buildbot
Date: Thu Oct 15 02:03:35 2015
New Revision: 968967
Log:
Staging update by buildbot for slider
Modified:
websites/staging/slider/trunk/content/ (props changed)
websites/staging/slider/trunk/content/docs/configuration/resources.html
Propchange: websites/staging/slider/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Thu Oct 15 02:03:35 2015
@@ -1 +1 @@
-1706153
+1708722
Modified: websites/staging/slider/trunk/content/docs/configuration/resources.html
==============================================================================
--- websites/staging/slider/trunk/content/docs/configuration/resources.html (original)
+++ websites/staging/slider/trunk/content/docs/configuration/resources.html Thu Oct 15 02:03:35 2015
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
<h1 class="title"></h1>
- <!---
+ <style type="text/css">
+/* The following code is added by mdx_elementid.py
+ It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+ visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink, h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink, dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -185,7 +196,7 @@ Latest release: <strong>0.80.0-incubatin
limitations under the License.
-->
-<h1 id="apache-slider-resource-specification">Apache Slider Resource Specification</h1>
+<h1 id="apache-slider-resource-specification">Apache Slider Resource Specification<a class="headerlink" href="#apache-slider-resource-specification" title="Permanent link">¶</a></h1>
<ul>
<li><a href="#core">Core Properties</a></li>
<li><a href="#failurepolicy">Container Failure Policy</a></li>
@@ -203,7 +214,7 @@ This includes:
these logs to be copied to HDFS and remotely retrieved, even while the application is running</p>
<p>As such, it is the core file used by Slider to configure and manage
the application.</p>
-<h2 id="core-properties"><a name="core"></a>Core Properties</h2>
+<h2 id="core-properties"><a name="core"></a>Core Properties<a class="headerlink" href="#core-properties" title="Permanent link">¶</a></h2>
<p>An example resource requirement for an application that has two components "master" and "worker" is as follows. Slider will automatically add the requirements for the AppMaster for the application. This component is named "slider-appmaster".</p>
<p>Some parameters that can be specified for a component instance include:</p>
<table>
@@ -234,7 +245,7 @@ the application.</p>
</table>
-<h3 id="component-instance-count-yarncomponentinstances">Component instance count: <code>yarn.component.instances</code></h3>
+<h3 id="component-instance-count-yarncomponentinstances">Component instance count: <code>yarn.component.instances</code><a class="headerlink" href="#component-instance-count-yarncomponentinstances" title="Permanent link">¶</a></h3>
<p>The property <code>yarn.component.instances</code> is one of the most foundational one in slider
âit declares how many instances of a component to instantiate on the cluster.</p>
<p>If the value is set to "0", no instances of a component will be created. If set
@@ -242,7 +253,7 @@ to a larger number, more instances will
of the application, component-by-component.</p>
<p>The number of instances of each component is application-specific; there are no recommended
values.</p>
-<h3 id="container-resource-requirements-yarnmemory-and-yarnvcores">Container resource requirements: <code>yarn.memory</code> and <code>yarn.vcores</code></h3>
+<h3 id="container-resource-requirements-yarnmemory-and-yarnvcores">Container resource requirements: <code>yarn.memory</code> and <code>yarn.vcores</code><a class="headerlink" href="#container-resource-requirements-yarnmemory-and-yarnvcores" title="Permanent link">¶</a></h3>
<p>These two properties define how much memory and CPU capacity each
YARN container of this component requires. YARN will queue
container requests until enough capacity exists within the cluster
@@ -278,14 +289,14 @@ time. </p>
has been granted to the container, it will simply be scheduled with less CPU time. The penalty
for using more CPU than requested is therefore less destructive than attempting to
use more memory than requested/granted.</p>
-<h4 id="relationship-between-yarnmemory-and-jvm-heap-size">Relationship between <code>yarn.memory</code> and JVM Heap Size</h4>
+<h4 id="relationship-between-yarnmemory-and-jvm-heap-size">Relationship between <code>yarn.memory</code> and JVM Heap Size<a class="headerlink" href="#relationship-between-yarnmemory-and-jvm-heap-size" title="Permanent link">¶</a></h4>
<p>Java applications deployed by Slider usually have a JVM heap size property which needs
to be defined as part of the application configuration.</p>
<p>The value of <code>yarn.memory</code> MUST be bigger than the heap size allocated to any JVM, as a JVM
uses a lot more memory than simply the heap alone. We have found that asking for at least 50%
more appears to work, though some experimentation will be needed. </p>
<p>Slider does not attempt to derive a heap size for any component from the YARN allocation.</p>
-<h3 id="component-instance-count-yarnrolepriority">Component instance count: <code>yarn.role.priority</code></h3>
+<h3 id="component-instance-count-yarnrolepriority">Component instance count: <code>yarn.role.priority</code><a class="headerlink" href="#component-instance-count-yarnrolepriority" title="Permanent link">¶</a></h3>
<p>The property <code>yarn.role.priority</code> has two purposes within Slider
1. It provides a unique index of individual component types. That is, it is not
the name of a component which Slider uses to index components, it is it's priority
@@ -299,7 +310,7 @@ request (defined by <code>yarn.component
each component's container defined by <code>yarn.memory</code> and <code>yarn.vcores</code>. It will request
the specified number of components âand keep those requests outstanding until they are
satisfied.</p>
-<h3 id="example">Example</h3>
+<h3 id="example">Example<a class="headerlink" href="#example" title="Permanent link">¶</a></h3>
<div class="codehilite"><pre><span class="p">{</span>
"<span class="n">schema</span>" <span class="p">:</span> "<span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">example</span><span class="p">.</span><span class="n">org</span><span class="o">/</span><span class="n">specification</span><span class="o">/</span><span class="n">v2</span><span class="p">.</span>0<span class="p">.</span>0"<span class="p">,</span>
"<span class="n">metadata</span>" <span class="p">:</span> <span class="p">{</span>
@@ -327,14 +338,14 @@ satisfied.</p>
</pre></div>
-<h2 id="the-slider-appmaster-component"><a name="slider-appmaster"></a>The <code>slider-appmaster</code> component</h2>
+<h2 id="the-slider-appmaster-component"><a name="slider-appmaster"></a>The <code>slider-appmaster</code> component<a class="headerlink" href="#the-slider-appmaster-component" title="Permanent link">¶</a></h2>
<p>The examples here all have a component <code>slider-appmaster</code>. This defines the settings of
the application master itself: the memory and CPU it requires, optionally a label (see
<a href="#labels">"Labels"</a>). The <code>yarn.role.priority</code> value is ignored: the priority is always "0";
and the instance count, <code>yarn.component.instances</code> is implicitly set to "1". </p>
<p>The entry exists primarily to allow applications to configure the amount of RAM the AM should
request.</p>
-<h2 id="container-failure-policy"><a name="failurepolicy"></a>Container Failure Policy</h2>
+<h2 id="container-failure-policy"><a name="failurepolicy"></a>Container Failure Policy<a class="headerlink" href="#container-failure-policy" title="Permanent link">¶</a></h2>
<p>YARN containers hosting component instances may fail. This can happen because of</p>
<ol>
<li>A problem in the configuration of the instance.</li>
@@ -348,8 +359,8 @@ and another running program.</li>
<p>Slider reacts to a failed container by requesting a new container from YARN,
preferably on a host that has already hosted an instance of that role. Once
the container is allocated, slider will redeploy an instance of the component.
-As it may time for YARN to have the resources to allocate the container, it
-may take some time for the replacement to be instantiated.</p>
+As it may take time for YARN to have the resources to allocate the container,
+replacements are not immediately instantiated.</p>
<p>Slider tracks failures in an attempt to differentiate problems in the
application package or its configuration from those of the underlying servers.
If a a component fails <em>too many times</em> then slider considers the application
@@ -359,13 +370,13 @@ itself as failing, and halts.</p>
1. The duration of a failure window, a time period in which failures are counted.
This duration can span days.
1. The maximum number of failures of any component in this time period.</p>
-<h3 id="failure-threshold-for-a-component">Failure threshold for a component</h3>
+<h3 id="failure-threshold-for-a-component">Failure threshold for a component<a class="headerlink" href="#failure-threshold-for-a-component" title="Permanent link">¶</a></h3>
<p>The number of times a component may fail within a failure window is
defined by the property <code>yarn.container.failure.threshold</code></p>
<p>If set to "0" there are no limits on
the number of times containers may fail.</p>
<p>The failure thresholds for individual components can be set independently</p>
-<h3 id="failure-window">Failure window</h3>
+<h3 id="failure-window">Failure window<a class="headerlink" href="#failure-window" title="Permanent link">¶</a></h3>
<p>The failure window can be set by minutes, days and hours. These must be set
in the <code>global</code> options, as they apply to slider only.</p>
<div class="codehilite"><pre><span class="n">yarn</span><span class="p">.</span><span class="n">container</span><span class="p">.</span><span class="n">failure</span><span class="p">.</span><span class="n">window</span><span class="p">.</span><span class="n">days</span>
@@ -388,7 +399,7 @@ restarted.</p>
<p>The default value is <code>yarn.container.failure.window.hours=6</code>; when changing
the window size, the hour value must be explicitly set, even if to zero, to
change this.</p>
-<h3 id="recommended-values">Recommended values</h3>
+<h3 id="recommended-values">Recommended values<a class="headerlink" href="#recommended-values" title="Permanent link">¶</a></h3>
<p>We recommend having a duration of a few hours for the window, and a
large failure limit proportional to the the number of instances of that
component </p>
@@ -397,7 +408,7 @@ component </p>
trying to reinstantiate all the components. Yet, if a component does fail
repeatedly, eventually slider will conclude that there is a problem and fail
with the exit code 73, <code>EXIT_DEPLOYMENT_FAILED</code>. </p>
-<h3 id="example_1">Example</h3>
+<h3 id="example_1">Example<a class="headerlink" href="#example_1" title="Permanent link">¶</a></h3>
<p>Here is a <code>resource.json</code> file for an HBase cluster:</p>
<div class="codehilite"><pre><span class="p">{</span>
"<span class="n">schema</span>" <span class="p">:</span> "<span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">example</span><span class="p">.</span><span class="n">org</span><span class="o">/</span><span class="n">specification</span><span class="o">/</span><span class="n">v2</span><span class="p">.</span>0<span class="p">.</span>0"<span class="p">,</span>
@@ -448,10 +459,10 @@ which are frequently failing due to conf
<p>In a production application, large failure thresholds and/or shorter windows
ensures that the application is resilient to transient failures of the underlying
YARN cluster and hardware.</p>
-<h2 id="placement-policies-and-escalation"><a name="placement"></a>Placement Policies and escalation</h2>
+<h2 id="placement-policies-and-escalation"><a name="placement"></a>Placement Policies and escalation<a class="headerlink" href="#placement-policies-and-escalation" title="Permanent link">¶</a></h2>
<p>Slider can be configured with different options for <strong>placement</strong> âthe
policies by which it chooses where to ask YARN for nodes.</p>
-<h3 id="placement-policy">Placement Policy</h3>
+<h3 id="placement-policy">Placement Policy<a class="headerlink" href="#placement-policy" title="Permanent link">¶</a></h3>
<p>The "placement policy" of a component is the set of rules by which Slider makes
a decision on where to request instances of that component from YARN.</p>
<table>
@@ -499,7 +510,7 @@ set in the property <code>"yarn.componen
<p>If the component were configured to request an explicit port for its REST endpoint,
then the same URL would reach it whenever this application were deployed
âprovided the host was available and the port not already in use.</p>
-<h4 id="notes">Notes</h4>
+<h4 id="notes">Notes<a class="headerlink" href="#notes" title="Permanent link">¶</a></h4>
<ol>
<li>
<p>There's no support for <strong>anti-affinity</strong> âi.e. to mandate that component
@@ -517,7 +528,7 @@ there.</p>
cannot be specified. The Application Master is deployed wherever YARN sees fit.</p>
</li>
</ol>
-<h3 id="node-failure-threshold-yarnnodefailurethreshold">Node Failure Threshold, <code>yarn.node.failure.threshold</code></h3>
+<h3 id="node-failure-threshold-yarnnodefailurethreshold">Node Failure Threshold, <code>yarn.node.failure.threshold</code><a class="headerlink" href="#node-failure-threshold-yarnnodefailurethreshold" title="Permanent link">¶</a></h3>
<p>The configuration property <code>yarn.node.failure.threshold</code> defines how "unreliable"
a node must be before it is skipped for placement requests.</p>
<ol>
@@ -526,7 +537,7 @@ a node must be before it is skipped for
<li>It is reset at the same time as the container failure counters; that is, at
the interval defined by the <code>yarn.container.failure.window</code> properties</li>
</ol>
-<h3 id="escalation-yarnplacementescalateseconds">Escalation: <code>yarn.placement.escalate.seconds</code></h3>
+<h3 id="escalation-yarnplacementescalateseconds">Escalation: <code>yarn.placement.escalate.seconds</code><a class="headerlink" href="#escalation-yarnplacementescalateseconds" title="Permanent link">¶</a></h3>
<p>For any component whose placement policy is not "any", Slider saves to HDFS
a record the nodes on which instances were running. When starting a cluster, it uses
this history to identify hosts on which to request instances. </p>
@@ -570,7 +581,7 @@ we would recommend for an escalation tim
</pre></div>
-<p>This declares that the <code>HBASE_MASTER</code> placement should be escalated after one second,
+<p>This declares that the <code>HBASE_MASTER</code> placement should be escalated after ten seconds,
but that that <code>HBASE_REGIONSERVER</code> instances should have an escalation timeout of 600
seconds âten minutes. These values were chosen as an HBase Master can be allocated
anywhere in the cluster, but a region server is significantly faster if restarted
@@ -578,10 +589,10 @@ on the same node on which it previously
have replicated all data elsewhere, it will have been scattered across the cluster
âresulting in remote access for most of the data, at least until a full compaction
has taken place.</p>
-<h4 id="notes_1">Notes</h4>
+<h4 id="notes_1">Notes<a class="headerlink" href="#notes_1" title="Permanent link">¶</a></h4>
<ol>
<li>
-<p>Escalation goes directory from "specific node" to "anywhere in cluster"; it does
+<p>Escalation goes directly from "specific node" to "anywhere in cluster"; it does
not have any intermediate "same-rack" policy.</p>
</li>
<li>
@@ -607,8 +618,8 @@ is not available or lacks capacity, the
<p>For more details, see: <a href="/design/rolehistory.html">Role History</a></p>
</li>
</ol>
-<h2 id="using-labels"><a name="labels"></a>Using Labels</h2>
-<p>The <code>resources.json</code> file include specifications the labels to be used when allocating containers for the components. The details of the YARN Label feature can be found at <a href="https://issues.apache.org/jira/browse/YARN-796">YARN-796</a>.</p>
+<h2 id="using-labels"><a name="labels"></a>Using Labels<a class="headerlink" href="#using-labels" title="Permanent link">¶</a></h2>
+<p>The <code>resources.json</code> file includes specifications of the labels to be used when allocating containers for the components. The details of the YARN Label feature can be found at <a href="https://issues.apache.org/jira/browse/YARN-796">YARN-796</a>.</p>
<p>In summary: </p>
<ul>
<li>Nodes can be assigned one or more labels</li>
@@ -619,7 +630,7 @@ is not available or lacks capacity, the
<p>This way, you can guarantee that a certain set of nodes are reserved for an application or for a component within an application.</p>
<p>Label expression is specified through property <code>yarn.label.expression</code>. When no label expression is specified then it is assumed that only non-labeled nodes are used when allocating containers for component instances.</p>
<p>If a label expression is specified for the <code>slider-appmaster</code> component then it also becomes the default label expression for all component. </p>
-<h4 id="example_2">Example</h4>
+<h4 id="example_2">Example<a class="headerlink" href="#example_2" title="Permanent link">¶</a></h4>
<p>Here is a <code>resource.json</code> file for an HBase cluster which uses labels.</p>
<p>The label for the application master is <code>hbase1</code> and the label expression for the HBASE_MASTER components is <code>hbase1_master</code>.
<code>HBASE_REGIONSERVER</code> instances will automatically use label <code>hbase1</code>.</p>
@@ -657,7 +668,7 @@ is not available or lacks capacity, the
<li>Perform refresh queue (<code>yarn -refreshqueue</code>) </li>
<li>Create the Slider application against the above queue using parameter --queue while creating the application</li>
</ol>
-<h3 id="notes_2">Notes</h3>
+<h3 id="notes_2">Notes<a class="headerlink" href="#notes_2" title="Permanent link">¶</a></h3>
<ol>
<li>
<p>If a label is defined in the <code>global</code> section, it will also apply to all components which do
@@ -681,7 +692,7 @@ any global- or appmaster- spettings, set
the application instance will not reach its requested size.</p>
</li>
</ol>
-<h2 id="log-aggregation"><a name="logagg"></a>Log Aggregation</h2>
+<h2 id="log-aggregation"><a name="logagg"></a>Log Aggregation<a class="headerlink" href="#log-aggregation" title="Permanent link">¶</a></h2>
<p>Log aggregation at regular intervals for long running services (LRS) needs to be enabled at the YARN level before
any application can exploit this functionality. To enable set <code>yarn.log-aggregation-enable</code> to true and the
interval property to a positive value of 3600 (in secs) or more. If set to a positive value less than 3600 (1 hour)
@@ -727,7 +738,7 @@ expressions. These properties need to be
<p>The details of the YARN Log Aggregation feature can be found at <a href="https://issues.apache.org/jira/browse/YARN-2468">YARN-2468</a>.</p>
-<h2 id="putting-it-all-together">Putting it all together</h2>
+<h2 id="putting-it-all-together">Putting it all together<a class="headerlink" href="#putting-it-all-together" title="Permanent link">¶</a></h2>
<p>Here is an example of a definition of an HBase cluster.</p>
<div class="codehilite"><pre><span class="p">{</span>
"<span class="n">schema</span>"<span class="p">:</span> "<span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">example</span><span class="p">.</span><span class="n">org</span><span class="o">/</span><span class="n">specification</span><span class="o">/</span><span class="n">v2</span><span class="p">.</span>0<span class="p">.</span>0"<span class="p">,</span>