You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@slider.apache.org by bu...@apache.org on 2014/11/12 08:47:03 UTC
svn commit: r928882 - in /websites/staging/slider/trunk/content: ./
docs/slider_specs/resource_specification.html
Author: buildbot
Date: Wed Nov 12 07:47:03 2014
New Revision: 928882
Log:
Staging update by buildbot for slider
Modified:
websites/staging/slider/trunk/content/ (props changed)
websites/staging/slider/trunk/content/docs/slider_specs/resource_specification.html
Propchange: websites/staging/slider/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Wed Nov 12 07:47:03 2014
@@ -1 +1 @@
-1638708
+1638709
Modified: websites/staging/slider/trunk/content/docs/slider_specs/resource_specification.html
==============================================================================
--- websites/staging/slider/trunk/content/docs/slider_specs/resource_specification.html (original)
+++ websites/staging/slider/trunk/content/docs/slider_specs/resource_specification.html Wed Nov 12 07:47:03 2014
@@ -215,7 +215,7 @@ Latest release: <strong>0.40-incubating<
</pre></div>
-<h2 id="wzxhzdk5wzxhzdk6container-failure-policy"><a name="failurepolicy"></a>Container Failure Policy</h2>
+<h2 id="wzxhzdk7wzxhzdk8container-failure-policy"><a name="failurepolicy"></a>Container Failure Policy</h2>
<p>YARN containers hosting component instances may fail. This can happen because of</p>
<ol>
<li>A problem in the configuration of the instance.</li>
@@ -318,7 +318,7 @@ which are frequently failing due to conf
<p>In a production application, large failure thresholds and/or shorter windows
ensures that the application is resilient to transient failures of the underlying
YARN cluster and hardware.</p>
-<h2 id="wzxhzdk7wzxhzdk8using-labels"><a name="labels"></a>Using Labels</h2>
+<h2 id="wzxhzdk9wzxhzdk10using-labels"><a name="labels"></a>Using Labels</h2>
<p>The resources.json file can be used to specify the labels to be used when allocating containers for the components. The details of the YARN Label feature can be found at <a href="https://issues.apache.org/jira/browse/YARN-796">YARN-796</a>.</p>
<p>In summary: </p>
<ul>
@@ -366,8 +366,47 @@ Here is a resource.json file for an HBas
<li>Perform refresh queue (yarn -refreshqueue) </li>
<li>Create the Slider application against the above queue using parameter --queue while creating the application</li>
</ul>
-<h2 id="wzxhzdk9wzxhzdk10using-log-aggregation"><a name="logagg"></a>Using Log Aggregation</h2>
-<p>The feature is not yet committed.</p>
+<h2 id="wzxhzdk11wzxhzdk12using-log-aggregation"><a name="logagg"></a>Using Log Aggregation</h2>
+<p>Log aggregation at regular intervals for long running services (LRS) needs to be enabled at the YARN level before
+any application can exploit this functionality. To enable set the following property to a positive value of 3600 (in secs)
+or more. If set to a positive value less than 3600 (1 hour) this property defaults to 3600. To disable log aggregation
+set it to -1.</p>
+<div class="codehilite"><pre> <span class="nt"><property></span>
+ <span class="nt"><name></span>yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds<span class="nt"></name></span>
+ <span class="nt"><value></span>3600<span class="nt"></value></span>
+ <span class="nt"></property></span>
+</pre></div>
+
+
+<p>Subsequently every application owner has the flexibility to set the include and exclude patterns of file names that
+they intent to aggregate. In Slider the resources.json file can be used to specify the include and exclude patterns
+of files that need to be backed up under the default log directory of the application. These properties needs to be
+set at the global level as shown below -</p>
+<div class="codehilite"><pre><span class="p">{</span>
+ "<span class="n">schema</span>"<span class="p">:</span> "<span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">example</span><span class="p">.</span><span class="n">org</span><span class="o">/</span><span class="n">specification</span><span class="o">/</span><span class="n">v2</span><span class="p">.</span>0<span class="p">.</span>0"<span class="p">,</span>
+ "<span class="n">metadata</span>"<span class="p">:</span> <span class="p">{</span>
+ <span class="p">},</span>
+ "<span class="k">global</span>"<span class="p">:</span> <span class="p">{</span>
+ "<span class="n">yarn</span><span class="p">.</span><span class="nb">log</span><span class="p">.</span><span class="n">include</span><span class="p">.</span><span class="n">patterns</span>"<span class="p">:</span> "<span class="n">hbase</span><span class="o">*.*</span>"<span class="p">,</span>
+ "<span class="n">yarn</span><span class="p">.</span><span class="nb">log</span><span class="p">.</span><span class="n">exclude</span><span class="p">.</span><span class="n">patterns</span>"<span class="p">:</span> "<span class="n">hbase</span><span class="o">*</span><span class="p">.</span><span class="n">out</span>"
+ <span class="p">},</span>
+ "<span class="n">components</span>"<span class="p">:</span> <span class="p">{</span>
+ "<span class="n">HBASE_MASTER</span>"<span class="p">:</span> <span class="p">{</span>
+ "<span class="n">yarn</span><span class="p">.</span><span class="n">role</span><span class="p">.</span><span class="n">priority</span>"<span class="p">:</span> "1"<span class="p">,</span>
+ "<span class="n">yarn</span><span class="p">.</span><span class="n">component</span><span class="p">.</span><span class="n">instances</span>"<span class="p">:</span> "1"<span class="p">,</span>
+ <span class="p">},</span>
+ "<span class="n">HBASE_REGIONSERVER</span>"<span class="p">:</span> <span class="p">{</span>
+ "<span class="n">yarn</span><span class="p">.</span><span class="n">role</span><span class="p">.</span><span class="n">priority</span>"<span class="p">:</span> "1"<span class="p">,</span>
+ "<span class="n">yarn</span><span class="p">.</span><span class="n">component</span><span class="p">.</span><span class="n">instances</span>"<span class="p">:</span> "1"<span class="p">,</span>
+ <span class="p">},</span>
+ "<span class="n">slider</span><span class="o">-</span><span class="n">appmaster</span>"<span class="p">:</span> <span class="p">{</span>
+ <span class="p">}</span>
+ <span class="p">}</span>
+<span class="p">}</span>
+</pre></div>
+
+
+<p>The details of the YARN Log Aggregation feature can be found at <a href="https://issues.apache.org/jira/browse/YARN-2468">YARN-2468</a>.</p>
</div>
<div id="footer">