You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@slider.apache.org by sm...@apache.org on 2014/07/08 02:49:35 UTC
svn commit: r1608633 [7/7] - in /incubator/slider/site: content/design/
content/design/registry/ content/design/specification/ content/developing/
content/docs/ content/docs/configuration/ content/docs/slider_specs/
trunk/content/developing/
Modified: incubator/slider/site/content/docs/slider_specs/application_package.html
URL: http://svn.apache.org/viewvc/incubator/slider/site/content/docs/slider_specs/application_package.html?rev=1608633&r1=1608632&r2=1608633&view=diff
==============================================================================
--- incubator/slider/site/content/docs/slider_specs/application_package.html (original)
+++ incubator/slider/site/content/docs/slider_specs/application_package.html Tue Jul 8 00:49:34 2014
@@ -231,34 +231,36 @@ other scripts, txt files, tarballs, etc.
</li>
</ol>
<p>The script specified in the metainfo is expected to understand the command. It can choose to call other scripts based on how the application author organizes the code base. For example:</p>
-<pre class="codehilite"><code>class OozieServer(Script):
- def install(self, env):
- self.install_packages(env)
-
- def configure(self, env):
- import params
- env.set_params(params)
- oozie(is_server=True)
-
- def start(self, env):
- import params
- env.set_params(params)
- self.configure(env)
- oozie_service(action='start')
-
- def stop(self, env):
- import params
- env.set_params(params)
- oozie_service(action='stop')
-
- def status(self, env):
- import status_params
- env.set_params(status_params)
- check_process_status(status_params.pid_file)</code></pre>
+<div class="codehilite"><pre><span class="n">class</span> <span class="n">OozieServer</span><span class="p">(</span><span class="n">Script</span><span class="p">)</span><span class="o">:</span>
+ <span class="n">def</span> <span class="n">install</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">env</span><span class="p">)</span><span class="o">:</span>
+ <span class="n">self</span><span class="p">.</span><span class="n">install_packages</span><span class="p">(</span><span class="n">env</span><span class="p">)</span>
+
+ <span class="n">def</span> <span class="n">configure</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">env</span><span class="p">)</span><span class="o">:</span>
+ <span class="n">import</span> <span class="n">params</span>
+ <span class="n">env</span><span class="p">.</span><span class="n">set_params</span><span class="p">(</span><span class="n">params</span><span class="p">)</span>
+ <span class="n">oozie</span><span class="p">(</span><span class="n">is_server</span><span class="o">=</span><span class="n">True</span><span class="p">)</span>
+
+ <span class="n">def</span> <span class="n">start</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">env</span><span class="p">)</span><span class="o">:</span>
+ <span class="n">import</span> <span class="n">params</span>
+ <span class="n">env</span><span class="p">.</span><span class="n">set_params</span><span class="p">(</span><span class="n">params</span><span class="p">)</span>
+ <span class="n">self</span><span class="p">.</span><span class="n">configure</span><span class="p">(</span><span class="n">env</span><span class="p">)</span>
+ <span class="n">oozie_service</span><span class="p">(</span><span class="n">action</span><span class="o">=</span><span class="err">'</span><span class="n">start</span><span class="err">'</span><span class="p">)</span>
+
+ <span class="n">def</span> <span class="n">stop</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">env</span><span class="p">)</span><span class="o">:</span>
+ <span class="n">import</span> <span class="n">params</span>
+ <span class="n">env</span><span class="p">.</span><span class="n">set_params</span><span class="p">(</span><span class="n">params</span><span class="p">)</span>
+ <span class="n">oozie_service</span><span class="p">(</span><span class="n">action</span><span class="o">=</span><span class="err">'</span><span class="n">stop</span><span class="err">'</span><span class="p">)</span>
+
+ <span class="n">def</span> <span class="n">status</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">env</span><span class="p">)</span><span class="o">:</span>
+ <span class="n">import</span> <span class="n">status_params</span>
+ <span class="n">env</span><span class="p">.</span><span class="n">set_params</span><span class="p">(</span><span class="n">status_params</span><span class="p">)</span>
+ <span class="n">check_process_status</span><span class="p">(</span><span class="n">status_params</span><span class="p">.</span><span class="n">pid_file</span><span class="p">)</span>
+</pre></div>
<p>The scripts are invoked in the following manner:</p>
-<pre class="codehilite"><code>python SCRIPT COMMAND JSON_FILE PACKAGE_ROOT STRUCTURED_OUT_FILE</code></pre>
+<div class="codehilite"><pre><span class="n">python</span> <span class="n">SCRIPT</span> <span class="n">COMMAND</span> <span class="n">JSON_FILE</span> <span class="n">PACKAGE_ROOT</span> <span class="n">STRUCTURED_OUT_FILE</span>
+</pre></div>
<ul>
@@ -296,11 +298,12 @@ other scripts, txt files, tarballs, etc.
</li>
</ul>
<p>Sample template file for dfs.exclude file to list excluded/decommissioned hosts. hdfs_exclude_files in the property defined in params.py which is populated from config parameters defined in JSON_FILE.</p>
-<pre class="codehilite"><code>{% if hdfs_exclude_file %}
-{% for host in hdfs_exclude_file %}
-{{host}}
-{% endfor %}
-{% endif %}</code></pre>
+<div class="codehilite"><pre><span class="cp">{%</span> <span class="k">if</span> <span class="nv">hdfs_exclude_file</span> <span class="cp">%}</span><span class="x"> </span>
+<span class="cp">{%</span> <span class="k">for</span> <span class="nv">host</span> <span class="k">in</span> <span class="nv">hdfs_exclude_file</span> <span class="cp">%}</span><span class="x"></span>
+<span class="cp">{{</span><span class="nv">host</span><span class="cp">}}</span><span class="x"></span>
+<span class="cp">{%</span> <span class="k">endfor</span> <span class="cp">%}</span><span class="x"></span>
+<span class="cp">{%</span> <span class="k">endif</span> <span class="cp">%}</span><span class="x"></span>
+</pre></div>
<h3 id="files-folder">files folder</h3>
Modified: incubator/slider/site/content/docs/slider_specs/creating_app_definitions.html
URL: http://svn.apache.org/viewvc/incubator/slider/site/content/docs/slider_specs/creating_app_definitions.html?rev=1608633&r1=1608632&r2=1608633&view=diff
==============================================================================
--- incubator/slider/site/content/docs/slider_specs/creating_app_definitions.html (original)
+++ incubator/slider/site/content/docs/slider_specs/creating_app_definitions.html Tue Jul 8 00:49:34 2014
@@ -176,15 +176,17 @@ Latest release: <strong>0.30-incubating<
<p><img alt="Image" src="../../resources/images/slider-container.png?raw=true" /></p>
<p>Figure 1 - High-level view of a container</p>
<p>For example:</p>
-<pre class="codehilite"><code>yarn 8849 -- python ./infra/agent/slider-agent/agent/main.py --label container_1397675825552_0011_01_000003___HBASE_REGIONSERVER --host AM_HOST --port 47830
-yarn 9085 -- bash /hadoop/yarn/local/usercache/yarn/appcache/application_1397675825552_0011/ ... internal_start regionserver
-yarn 9114 -- /usr/jdk64/jdk1.7.0_45/bin/java -Dproc_regionserver -XX:OnOutOfMemoryError=...</code></pre>
+<div class="codehilite"><pre><span class="n">yarn</span> <span class="mi">8849</span> <span class="o">--</span> <span class="n">python</span> <span class="p">.</span><span class="o">/</span><span class="n">infra</span><span class="o">/</span><span class="n">agent</span><span class="o">/</span><span class="n">slider</span><span class="o">-</span><span class="n">agent</span><span class="o">/</span><span class="n">agent</span><span class="o">/</span><span class="n">main</span><span class="p">.</span><span class="n">py</span> <span class="o">--</span><span class="n">label</span> <span class="n">container_1397675825552_0011_01_000003___HBASE_REGIONSERVER</span> <span class="o">--</span><span class="n">host</span> <span class="n">AM_HOST</span> <span class="o">--</span><span class="n">port</span> <span class="mi">47830</span>
+<span class="n">yarn</span> <span class="mi">9085</span> <span class="o">--</span> <span class="n">bash</span> <span class="o">/</span><span class="n">hadoop</span><span class="o">/</span><span class="n">yarn</span><span class="o">/</span><span class="n">local</span><span class="o">/</span><span class="n">usercache</span><span class="o">/</span><span class="n">yarn</span><span class="o">/</span><span class="n">appcache</span><span class="o">/</span><span class="n">application_1397675825552_0011</span><span class="o">/</span> <span class="p">...</span> <span class="n">internal_start</span> <span class="n">regionserver</span>
+<span class="n">yarn</span> <span class="mi">9114</span> <span class="o">--</span> <span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">jdk64</span><span class="o">/</span><span class="n">jdk1</span><span class="mf">.7.0</span><span class="n">_45</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">java</span> <span class="o">-</span><span class="n">Dproc_regionserver</span> <span class="o">-</span><span class="n">XX</span><span class="o">:</span><span class="n">OnOutOfMemoryError</span><span class="o">=</span><span class="p">...</span>
+</pre></div>
<p>The above list shows three processes, the Slider-Agent process, the bash script to start HBase Region Server and the HBase Region server itself. <em>Three of these together constitute the container</em>. </p>
<h2 id="using-an-apppackage">Using an AppPackage</h2>
<p>The following command creates an HBase application using the AppPackage for HBase.</p>
-<pre class="codehilite"><code> ./slider create cl1 --image hdfs://NN:8020/slider/agent/slider-agent.tar.gz --template /work/appConf.json --resources /work/resources.json</code></pre>
+<div class="codehilite"><pre> <span class="p">.</span><span class="o">/</span><span class="n">slider</span> <span class="n">create</span> <span class="n">cl1</span> <span class="o">--</span><span class="n">image</span> <span class="n">hdfs</span><span class="o">:</span><span class="c1">//NN:8020/slider/agent/slider-agent.tar.gz --template /work/appConf.json --resources /work/resources.json</span>
+</pre></div>
<p>Lets analyze various parameters from the perspective of app creation:</p>
@@ -210,7 +212,8 @@ yarn 9114 -- /usr/jdk64/jdk1.7.0_45
</ul>
<p><strong>They are placehoder files</strong>, mostly because the files themselves are too large as well as users are free to use their own version of the package. To create a Slider AppPackage - replace the file with an actual application tarball and then ensure that the metainfo.xml has the correct file name. After that create a zip file using standard zip commands and ensure that the package has the metainfo.xml file at the root folder.</p>
<p>For Accumulo, this is automated in maven by running</p>
-<pre class="codehilite"><code>mvn clean package -DskipTests -Paccumulo-app-package</code></pre>
+<div class="codehilite"><pre><span class="n">mvn</span> <span class="n">clean</span> <span class="n">package</span> <span class="o">-</span><span class="n">DskipTests</span> <span class="o">-</span><span class="n">Paccumulo</span><span class="o">-</span><span class="n">app</span><span class="o">-</span><span class="n">package</span>
+</pre></div>
<p>Additional configuration parameters are described in <code>app-packages/accumulo/README.txt</code>.</p>
Modified: incubator/slider/site/content/docs/slider_specs/resource_specification.html
URL: http://svn.apache.org/viewvc/incubator/slider/site/content/docs/slider_specs/resource_specification.html?rev=1608633&r1=1608632&r2=1608633&view=diff
==============================================================================
--- incubator/slider/site/content/docs/slider_specs/resource_specification.html (original)
+++ incubator/slider/site/content/docs/slider_specs/resource_specification.html Tue Jul 8 00:49:34 2014
@@ -181,27 +181,28 @@ Latest release: <strong>0.30-incubating<
<li><code>yarn.component.instances</code>: number of instances for this component type</li>
</ul>
<p>Sample:</p>
-<pre class="codehilite"><code>{
- "schema" : "http://example.org/specification/v2.0.0",
- "metadata" : {
- },
- "global" : {
- },
- "components" : {
- "HBASE_MASTER" : {
- "yarn.role.priority" : "1",
- "yarn.component.instances" : "1"
- "yarn.memory" : "768",
- "yarn.vcores" : "1"
- },
- "slider-appmaster" : {
- },
- "HBASE_REGIONSERVER" : {
- "yarn.role.priority" : "2",
- "yarn.component.instances" : "1"
- }
- }
-}</code></pre>
+<div class="codehilite"><pre><span class="p">{</span>
+ <span class="s">"schema"</span> <span class="o">:</span> <span class="s">"http://example.org/specification/v2.0.0"</span><span class="p">,</span>
+ <span class="s">"metadata"</span> <span class="o">:</span> <span class="p">{</span>
+ <span class="p">},</span>
+ <span class="s">"global"</span> <span class="o">:</span> <span class="p">{</span>
+ <span class="p">},</span>
+ <span class="s">"components"</span> <span class="o">:</span> <span class="p">{</span>
+ <span class="s">"HBASE_MASTER"</span> <span class="o">:</span> <span class="p">{</span>
+ <span class="s">"yarn.role.priority"</span> <span class="o">:</span> <span class="s">"1"</span><span class="p">,</span>
+ <span class="s">"yarn.component.instances"</span> <span class="o">:</span> <span class="s">"1"</span>
+ <span class="s">"yarn.memory"</span> <span class="o">:</span> <span class="s">"768"</span><span class="p">,</span>
+ <span class="s">"yarn.vcores"</span> <span class="o">:</span> <span class="s">"1"</span>
+ <span class="p">},</span>
+ <span class="s">"slider-appmaster"</span> <span class="o">:</span> <span class="p">{</span>
+ <span class="p">},</span>
+ <span class="s">"HBASE_REGIONSERVER"</span> <span class="o">:</span> <span class="p">{</span>
+ <span class="s">"yarn.role.priority"</span> <span class="o">:</span> <span class="s">"2"</span><span class="p">,</span>
+ <span class="s">"yarn.component.instances"</span> <span class="o">:</span> <span class="s">"1"</span>
+ <span class="p">}</span>
+ <span class="p">}</span>
+<span class="p">}</span>
+</pre></div>
</div>
<div id="footer">
Modified: incubator/slider/site/content/docs/slider_specs/writing_app_command_scripts.html
URL: http://svn.apache.org/viewvc/incubator/slider/site/content/docs/slider_specs/writing_app_command_scripts.html?rev=1608633&r1=1608632&r2=1608633&view=diff
==============================================================================
--- incubator/slider/site/content/docs/slider_specs/writing_app_command_scripts.html (original)
+++ incubator/slider/site/content/docs/slider_specs/writing_app_command_scripts.html Tue Jul 8 00:49:34 2014
@@ -172,7 +172,8 @@ Latest release: <strong>0.30-incubating<
<h1 id="apache-slider-developing-app-command-scripts">Apache Slider: Developing App Command Scripts</h1>
<p>App command implementations follow a standard structure so that they can be invoked in an uniform manner. For any command, the python scripts are invoked as:</p>
-<pre class="codehilite"><code>python SCRIPT COMMAND JSON_FILE PACKAGE_ROOT STRUCTURED_OUT_FILE</code></pre>
+<div class="codehilite"><pre><span class="n">python</span> <span class="n">SCRIPT</span> <span class="n">COMMAND</span> <span class="n">JSON_FILE</span> <span class="n">PACKAGE_ROOT</span> <span class="n">STRUCTURED_OUT_FILE</span>
+</pre></div>
<ul>
@@ -194,43 +195,45 @@ Latest release: <strong>0.30-incubating<
</ul>
<p>The management infrastructure is expected to automatically reports back STD_OUT and STD_ERR.</p>
<p>Sample:</p>
-<pre class="codehilite"><code>python /apps/HBASE_ON_YARN/package/scripts/hbase_regionserver.py START /apps/commands/cmd_332/command.json /apps/HBASE_ON_YARN/package /apps/commands/cmd_332/strout.txt</code></pre>
+<div class="codehilite"><pre><span class="n">python</span> <span class="o">/</span><span class="n">apps</span><span class="o">/</span><span class="n">HBASE_ON_YARN</span><span class="o">/</span><span class="n">package</span><span class="o">/</span><span class="n">scripts</span><span class="o">/</span><span class="n">hbase_regionserver</span><span class="p">.</span><span class="n">py</span> <span class="n">START</span> <span class="o">/</span><span class="n">apps</span><span class="o">/</span><span class="n">commands</span><span class="o">/</span><span class="n">cmd_332</span><span class="o">/</span><span class="n">command</span><span class="p">.</span><span class="n">json</span> <span class="o">/</span><span class="n">apps</span><span class="o">/</span><span class="n">HBASE_ON_YARN</span><span class="o">/</span><span class="n">package</span> <span class="o">/</span><span class="n">apps</span><span class="o">/</span><span class="n">commands</span><span class="o">/</span><span class="
n">cmd_332</span><span class="o">/</span><span class="n">strout</span><span class="p">.</span><span class="n">txt</span>
+</pre></div>
<p><strong>Note</strong>: The above is how Slider-Agent invokes the scripts. Its provided as a reference for developing the scripts themselves as well as a way to test/debug the scripts.</p>
<h2 id="structure-of-json-formatted-parameter">Structure of JSON formatted parameter</h2>
<p>The parameters are organized as multi-layer name-value pairs.</p>
-<pre class="codehilite"><code>{
- "commandId": "Command Id as assigned by Slider",
- "command": "Command being executed",
- "commandType": "Type of command",
- "clusterName": "Name of the cluster",
- "appName": "Name of the app",
- "component": "Name of the component",
- "hostname": "Name of the host",
- "public_hostname": "FQDN of the host",
- "hostParams": {
- "host specific parameters common to all commands"
- },
- "componentParams": {
- "component specific parameters, if any"
- },
- "commandParams": {
- "command specific parameters, usually used in case of custom commands"
- },
- "configurations": {
- "app-global-config": {
- },
- "config-type-2": {
- },
- "config-type-2": {
- }
- }
-}</code></pre>
+<div class="codehilite"><pre><span class="p">{</span>
+ <span class="s">"commandId"</span><span class="o">:</span> <span class="s">"Command Id as assigned by Slider"</span><span class="p">,</span>
+ <span class="s">"command"</span><span class="o">:</span> <span class="s">"Command being executed"</span><span class="p">,</span>
+ <span class="s">"commandType"</span><span class="o">:</span> <span class="s">"Type of command"</span><span class="p">,</span>
+ <span class="s">"clusterName"</span><span class="o">:</span> <span class="s">"Name of the cluster"</span><span class="p">,</span>
+ <span class="s">"appName"</span><span class="o">:</span> <span class="s">"Name of the app"</span><span class="p">,</span>
+ <span class="s">"component"</span><span class="o">:</span> <span class="s">"Name of the component"</span><span class="p">,</span>
+ <span class="s">"hostname"</span><span class="o">:</span> <span class="s">"Name of the host"</span><span class="p">,</span>
+ <span class="s">"public_hostname"</span><span class="o">:</span> <span class="s">"FQDN of the host"</span><span class="p">,</span>
+ <span class="s">"hostParams"</span><span class="o">:</span> <span class="p">{</span>
+ <span class="s">"host specific parameters common to all commands"</span>
+ <span class="p">},</span>
+ <span class="s">"componentParams"</span><span class="o">:</span> <span class="p">{</span>
+ <span class="s">"component specific parameters, if any"</span>
+ <span class="p">},</span>
+ <span class="s">"commandParams"</span><span class="o">:</span> <span class="p">{</span>
+ <span class="s">"command specific parameters, usually used in case of custom commands"</span>
+ <span class="p">},</span>
+ <span class="s">"configurations"</span><span class="o">:</span> <span class="p">{</span>
+ <span class="s">"app-global-config"</span><span class="o">:</span> <span class="p">{</span>
+ <span class="p">},</span>
+ <span class="s">"config-type-2"</span><span class="o">:</span> <span class="p">{</span>
+ <span class="p">},</span>
+ <span class="s">"config-type-2"</span><span class="o">:</span> <span class="p">{</span>
+ <span class="p">}</span>
+ <span class="p">}</span>
+<span class="p">}</span>
+</pre></div>
<h2 id="sample-configuration-parameters">Sample configuration parameters</h2>
-<pre class="codehilite"><code>{
+<div class="codehilite"><pre>{
"commandId": "2-2",
"command": "START",
"commandType": "EXECUTION_COMMAND",
@@ -247,12 +250,12 @@ Latest release: <strong>0.30-incubating<
"configurations": {
"hbase-log4j": {
"log4j.threshold": "ALL",
- "log4j.rootLogger": "${hbase.root.logger}",
+ "log4j.rootLogger": "<span class="cp">${</span><span class="n">hbase</span><span class="o">.</span><span class="n">root</span><span class="o">.</span><span class="n">logger</span><span class="cp">}</span>",
"log4j.logger.org.apache.zookeeper": "INFO",
"log4j.logger.org.apache.hadoop.hbase": "DEBUG",
"log4j.logger.org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher": "INFO",
"log4j.logger.org.apache.hadoop.hbase.zookeeper.ZKUtil": "INFO",
- "log4j.category.SecurityLogger": "${hbase.security.logger}",
+ "log4j.category.SecurityLogger": "<span class="cp">${</span><span class="n">hbase</span><span class="o">.</span><span class="n">security</span><span class="o">.</span><span class="n">logger</span><span class="cp">}</span>",
"log4j.appender.console": "org.apache.log4j.ConsoleAppender",
"log4j.appender.console.target": "System.err",
"log4j.appender.console.layout": "org.apache.log4j.PatternLayout",
@@ -260,20 +263,20 @@ Latest release: <strong>0.30-incubating<
"log4j.appender.RFAS": "org.apache.log4j.RollingFileAppender",
"log4j.appender.RFAS.layout": "org.apache.log4j.PatternLayout",
"log4j.appender.RFAS.layout.ConversionPattern": "%d{ISO8601} %p %c: %m%n",
- "log4j.appender.RFAS.MaxFileSize": "${hbase.security.log.maxfilesize}",
- "log4j.appender.RFAS.MaxBackupIndex": "${hbase.security.log.maxbackupindex}",
- "log4j.appender.RFAS.File": "${hbase.log.dir}/${hbase.security.log.file}",
+ "log4j.appender.RFAS.MaxFileSize": "<span class="cp">${</span><span class="n">hbase</span><span class="o">.</span><span class="n">security</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">maxfilesize</span><span class="cp">}</span>",
+ "log4j.appender.RFAS.MaxBackupIndex": "<span class="cp">${</span><span class="n">hbase</span><span class="o">.</span><span class="n">security</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">maxbackupindex</span><span class="cp">}</span>",
+ "log4j.appender.RFAS.File": "<span class="cp">${</span><span class="n">hbase</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">dir</span><span class="cp">}</span>/<span class="cp">${</span><span class="n">hbase</span><span class="o">.</span><span class="n">security</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">file</span><span class="cp">}</span>",
"log4j.appender.RFA": "org.apache.log4j.RollingFileAppender",
"log4j.appender.RFA.layout": "org.apache.log4j.PatternLayout",
"log4j.appender.RFA.layout.ConversionPattern": "%d{ISO8601} %-5p [%t] %c{2}: %m%n",
- "log4j.appender.RFA.MaxFileSize": "${hbase.log.maxfilesize}",
- "log4j.appender.RFA.MaxBackupIndex": "${hbase.log.maxbackupindex}",
- "log4j.appender.RFA.File": "${hbase.log.dir}/${hbase.log.file}",
+ "log4j.appender.RFA.MaxFileSize": "<span class="cp">${</span><span class="n">hbase</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">maxfilesize</span><span class="cp">}</span>",
+ "log4j.appender.RFA.MaxBackupIndex": "<span class="cp">${</span><span class="n">hbase</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">maxbackupindex</span><span class="cp">}</span>",
+ "log4j.appender.RFA.File": "<span class="cp">${</span><span class="n">hbase</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">dir</span><span class="cp">}</span>/<span class="cp">${</span><span class="n">hbase</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">file</span><span class="cp">}</span>",
"log4j.appender.NullAppender": "org.apache.log4j.varia.NullAppender",
"log4j.appender.DRFA": "org.apache.log4j.DailyRollingFileAppender",
"log4j.appender.DRFA.layout": "org.apache.log4j.PatternLayout",
"log4j.appender.DRFA.layout.ConversionPattern": "%d{ISO8601} %-5p [%t] %c{2}: %m%n",
- "log4j.appender.DRFA.File": "${hbase.log.dir}/${hbase.log.file}",
+ "log4j.appender.DRFA.File": "<span class="cp">${</span><span class="n">hbase</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">dir</span><span class="cp">}</span>/<span class="cp">${</span><span class="n">hbase</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">file</span><span class="cp">}</span>",
"log4j.appender.DRFA.DatePattern": ".yyyy-MM-dd",
"log4j.additivity.SecurityLogger": "false",
"hbase.security.logger": "INFO,console",
@@ -333,34 +336,36 @@ Latest release: <strong>0.30-incubating<
"hbase.master.info.port": "60010"
}
}
-}</code></pre>
+}
+</pre></div>
<h2 id="sample-command-script">Sample command script</h2>
-<pre class="codehilite"><code>class OozieServer(Script):
- def install(self, env):
- self.install_packages(env)
-
- def configure(self, env):
- import params
- env.set_params(params)
- oozie(is_server=True)
-
- def start(self, env):
- import params
- env.set_params(params)
- self.configure(env)
- oozie_service(action='start')
-
- def stop(self, env):
- import params
- env.set_params(params)
- oozie_service(action='stop')
-
- def status(self, env):
- import status_params
- env.set_params(status_params)
- check_process_status(status_params.pid_file)</code></pre>
+<div class="codehilite"><pre><span class="n">class</span> <span class="n">OozieServer</span><span class="p">(</span><span class="n">Script</span><span class="p">)</span><span class="o">:</span>
+ <span class="n">def</span> <span class="n">install</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">env</span><span class="p">)</span><span class="o">:</span>
+ <span class="n">self</span><span class="p">.</span><span class="n">install_packages</span><span class="p">(</span><span class="n">env</span><span class="p">)</span>
+
+ <span class="n">def</span> <span class="n">configure</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">env</span><span class="p">)</span><span class="o">:</span>
+ <span class="n">import</span> <span class="n">params</span>
+ <span class="n">env</span><span class="p">.</span><span class="n">set_params</span><span class="p">(</span><span class="n">params</span><span class="p">)</span>
+ <span class="n">oozie</span><span class="p">(</span><span class="n">is_server</span><span class="o">=</span><span class="n">True</span><span class="p">)</span>
+
+ <span class="n">def</span> <span class="n">start</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">env</span><span class="p">)</span><span class="o">:</span>
+ <span class="n">import</span> <span class="n">params</span>
+ <span class="n">env</span><span class="p">.</span><span class="n">set_params</span><span class="p">(</span><span class="n">params</span><span class="p">)</span>
+ <span class="n">self</span><span class="p">.</span><span class="n">configure</span><span class="p">(</span><span class="n">env</span><span class="p">)</span>
+ <span class="n">oozie_service</span><span class="p">(</span><span class="n">action</span><span class="o">=</span><span class="err">'</span><span class="n">start</span><span class="err">'</span><span class="p">)</span>
+
+ <span class="n">def</span> <span class="n">stop</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">env</span><span class="p">)</span><span class="o">:</span>
+ <span class="n">import</span> <span class="n">params</span>
+ <span class="n">env</span><span class="p">.</span><span class="n">set_params</span><span class="p">(</span><span class="n">params</span><span class="p">)</span>
+ <span class="n">oozie_service</span><span class="p">(</span><span class="n">action</span><span class="o">=</span><span class="err">'</span><span class="n">stop</span><span class="err">'</span><span class="p">)</span>
+
+ <span class="n">def</span> <span class="n">status</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">env</span><span class="p">)</span><span class="o">:</span>
+ <span class="n">import</span> <span class="n">status_params</span>
+ <span class="n">env</span><span class="p">.</span><span class="n">set_params</span><span class="p">(</span><span class="n">status_params</span><span class="p">)</span>
+ <span class="n">check_process_status</span><span class="p">(</span><span class="n">status_params</span><span class="p">.</span><span class="n">pid_file</span><span class="p">)</span>
+</pre></div>
</div>
<div id="footer">
Modified: incubator/slider/site/content/docs/troubleshooting.html
URL: http://svn.apache.org/viewvc/incubator/slider/site/content/docs/troubleshooting.html?rev=1608633&r1=1608632&r2=1608633&view=diff
==============================================================================
--- incubator/slider/site/content/docs/troubleshooting.html (original)
+++ incubator/slider/site/content/docs/troubleshooting.html Tue Jul 8 00:49:34 2014
@@ -180,24 +180,26 @@ that works</p>
those JARs it has locally, and the JARS pre-installed on the classpath</p>
<p>This often surfaces in an exception that can be summarized as
"hadoop-common.jar is not on the classpath":</p>
-<pre class="codehilite"><code>Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/ExitUtil$ExitException
-Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.ExitUtil$ExitException
- at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
- at java.security.AccessController.doPrivileged(Native Method)
- at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
- at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
- at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
- at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
-Could not find the main class: org.apache.hadoop.yarn.service.launcher.ServiceLauncher. Program will exit.</code></pre>
+<div class="codehilite"><pre><span class="n">Exception</span> <span class="n">in</span> <span class="kr">thread</span> <span class="s">"main"</span> <span class="n">java</span><span class="p">.</span><span class="n">lang</span><span class="p">.</span><span class="n">NoClassDefFoundError</span><span class="o">:</span> <span class="n">org</span><span class="o">/</span><span class="n">apache</span><span class="o">/</span><span class="n">hadoop</span><span class="o">/</span><span class="n">util</span><span class="o">/</span><span class="n">ExitUtil</span><span class="err">$</span><span class="n">ExitException</span>
+<span class="n">Caused</span> <span class="n">by</span><span class="o">:</span> <span class="n">java</span><span class="p">.</span><span class="n">lang</span><span class="p">.</span><span class="n">ClassNotFoundException</span><span class="o">:</span> <span class="n">org</span><span class="p">.</span><span class="n">apache</span><span class="p">.</span><span class="n">hadoop</span><span class="p">.</span><span class="n">util</span><span class="p">.</span><span class="n">ExitUtil</span><span class="err">$</span><span class="n">ExitException</span>
+ <span class="n">at</span> <span class="n">java</span><span class="p">.</span><span class="n">net</span><span class="p">.</span><span class="n">URLClassLoader</span><span class="err">$</span><span class="mf">1.</span><span class="n">run</span><span class="p">(</span><span class="n">URLClassLoader</span><span class="p">.</span><span class="n">java</span><span class="o">:</span><span class="mi">202</span><span class="p">)</span>
+ <span class="n">at</span> <span class="n">java</span><span class="p">.</span><span class="n">security</span><span class="p">.</span><span class="n">AccessController</span><span class="p">.</span><span class="n">doPrivileged</span><span class="p">(</span><span class="n">Native</span> <span class="n">Method</span><span class="p">)</span>
+ <span class="n">at</span> <span class="n">java</span><span class="p">.</span><span class="n">net</span><span class="p">.</span><span class="n">URLClassLoader</span><span class="p">.</span><span class="n">findClass</span><span class="p">(</span><span class="n">URLClassLoader</span><span class="p">.</span><span class="n">java</span><span class="o">:</span><span class="mi">190</span><span class="p">)</span>
+ <span class="n">at</span> <span class="n">java</span><span class="p">.</span><span class="n">lang</span><span class="p">.</span><span class="n">ClassLoader</span><span class="p">.</span><span class="n">loadClass</span><span class="p">(</span><span class="n">ClassLoader</span><span class="p">.</span><span class="n">java</span><span class="o">:</span><span class="mi">306</span><span class="p">)</span>
+ <span class="n">at</span> <span class="n">sun</span><span class="p">.</span><span class="n">misc</span><span class="p">.</span><span class="n">Launcher</span><span class="err">$</span><span class="n">AppClassLoader</span><span class="p">.</span><span class="n">loadClass</span><span class="p">(</span><span class="n">Launcher</span><span class="p">.</span><span class="n">java</span><span class="o">:</span><span class="mi">301</span><span class="p">)</span>
+ <span class="n">at</span> <span class="n">java</span><span class="p">.</span><span class="n">lang</span><span class="p">.</span><span class="n">ClassLoader</span><span class="p">.</span><span class="n">loadClass</span><span class="p">(</span><span class="n">ClassLoader</span><span class="p">.</span><span class="n">java</span><span class="o">:</span><span class="mi">247</span><span class="p">)</span>
+<span class="n">Could</span> <span class="n">not</span> <span class="n">find</span> <span class="n">the</span> <span class="n">main</span> <span class="n">class</span><span class="o">:</span> <span class="n">org</span><span class="p">.</span><span class="n">apache</span><span class="p">.</span><span class="n">hadoop</span><span class="p">.</span><span class="n">yarn</span><span class="p">.</span><span class="n">service</span><span class="p">.</span><span class="n">launcher</span><span class="p">.</span><span class="n">ServiceLauncher</span><span class="p">.</span> <span class="n">Program</span> <span class="n">will</span> <span class="n">exit</span><span class="p">.</span>
+</pre></div>
<p>For ambari-managed deployments, we recommend the following</p>
-<pre class="codehilite"><code> <property>
- <name>yarn.application.classpath</name>
- <value>
+<div class="codehilite"><pre> <span class="nt"><property></span>
+ <span class="nt"><name></span>yarn.application.classpath<span class="nt"></name></span>
+ <span class="nt"><value></span>
/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*
- </value>
- </property></code></pre>
+ <span class="nt"></value></span>
+ <span class="nt"></property></span>
+</pre></div>
<p>The <code>yarn-site.xml</code> file for the site will contain the relevant value.</p>
@@ -205,10 +207,11 @@ Could not find the main class: org.apach
<p>Slider gives up if it cannot keep enough instances of a role running -or more
precisely, if they keep failing. </p>
<p>If this happens on cluster startup, it means that the application is not working</p>
-<pre class="codehilite"><code> org.apache.slider.core.exceptions.TriggerClusterTeardownException: Unstable Cluster:
- - failed with role worker failing 4 times (4 in startup); threshold is 2
- - last failure: Failure container_1386872971874_0001_01_000006 on host 192.168.1.86,
- see http://hor12n22.gq1.ygridcore.net:19888/jobhistory/logs/192.168.1.86:45454/container_1386872971874_0001_01_000006/ctx/yarn</code></pre>
+<div class="codehilite"><pre> <span class="n">org</span><span class="p">.</span><span class="n">apache</span><span class="p">.</span><span class="n">slider</span><span class="p">.</span><span class="n">core</span><span class="p">.</span><span class="n">exceptions</span><span class="p">.</span><span class="n">TriggerClusterTeardownException</span><span class="o">:</span> <span class="n">Unstable</span> <span class="n">Cluster</span><span class="o">:</span>
+ <span class="o">-</span> <span class="n">failed</span> <span class="n">with</span> <span class="n">role</span> <span class="n">worker</span> <span class="n">failing</span> <span class="mi">4</span> <span class="n">times</span> <span class="p">(</span><span class="mi">4</span> <span class="n">in</span> <span class="n">startup</span><span class="p">);</span> <span class="n">threshold</span> <span class="n">is</span> <span class="mi">2</span>
+ <span class="o">-</span> <span class="n">last</span> <span class="n">failure</span><span class="o">:</span> <span class="n">Failure</span> <span class="n">container_1386872971874_0001_01_000006</span> <span class="n">on</span> <span class="n">host</span> <span class="mf">192.168.1.86</span><span class="p">,</span>
+ <span class="n">see</span> <span class="n">http</span><span class="o">:</span><span class="c1">//hor12n22.gq1.ygridcore.net:19888/jobhistory/logs/192.168.1.86:45454/container_1386872971874_0001_01_000006/ctx/yarn</span>
+</pre></div>
<p>This message warns that a role -here worker- is failing to start and it has failed
@@ -237,22 +240,23 @@ application component deployed</em></p>
<p>Fix #2: Tell YARN to be less strict about memory consumption</p>
<p>Here are the properties in <code>yarn-site.xml</code> which we set to allow YARN
to schedule more role instances than it nominally has room for.</p>
-<pre class="codehilite"><code><property>
- <name>yarn.scheduler.minimum-allocation-mb</name>
- <value>1</value>
-</property>
-<property>
- <description>Whether physical memory limits will be enforced for
+<div class="codehilite"><pre><span class="nt"><property></span>
+ <span class="nt"><name></span>yarn.scheduler.minimum-allocation-mb<span class="nt"></name></span>
+ <span class="nt"><value></span>1<span class="nt"></value></span>
+<span class="nt"></property></span>
+<span class="nt"><property></span>
+ <span class="nt"><description></span>Whether physical memory limits will be enforced for
containers.
- </description>
- <name>yarn.nodemanager.pmem-check-enabled</name>
- <value>false</value>
-</property>
-<!-- we really don't want checking here-->
-<property>
- <name>yarn.nodemanager.vmem-check-enabled</name>
- <value>false</value>
-</property></code></pre>
+ <span class="nt"></description></span>
+ <span class="nt"><name></span>yarn.nodemanager.pmem-check-enabled<span class="nt"></name></span>
+ <span class="nt"><value></span>false<span class="nt"></value></span>
+<span class="nt"></property></span>
+<span class="c"><!-- we really don't want checking here--></span>
+<span class="nt"><property></span>
+ <span class="nt"><name></span>yarn.nodemanager.vmem-check-enabled<span class="nt"></name></span>
+ <span class="nt"><value></span>false<span class="nt"></value></span>
+<span class="nt"></property></span>
+</pre></div>
<p>If you create too many instances, your hosts will start swapping and
@@ -260,21 +264,23 @@ performance will collapse -we do not rec
<h3 id="configuring-yarn-for-better-debugging">Configuring YARN for better debugging</h3>
<p>One configuration to aid debugging is tell the nodemanagers to
keep data for a short period after containers finish</p>
-<pre class="codehilite"><code><!-- 10 minutes after a failure to see what is left in the directory-->
-<property>
- <name>yarn.nodemanager.delete.debug-delay-sec</name>
- <value>600</value>
-</property></code></pre>
+<div class="codehilite"><pre><span class="c"><!-- 10 minutes after a failure to see what is left in the directory--></span>
+<span class="nt"><property></span>
+ <span class="nt"><name></span>yarn.nodemanager.delete.debug-delay-sec<span class="nt"></name></span>
+ <span class="nt"><value></span>600<span class="nt"></value></span>
+<span class="nt"></property></span>
+</pre></div>
<p>You can then retrieve logs by either the web UI, or by connecting to the
server (usually by <code>ssh</code>) and retrieve the logs from the log directory</p>
<p>We also recommend making sure that YARN kills processes</p>
-<pre class="codehilite"><code><!--time before the process gets a -9 -->
-<property>
- <name>yarn.nodemanager.sleep-delay-before-sigkill.ms</name>
- <value>30000</value>
-</property></code></pre>
+<div class="codehilite"><pre><span class="c"><!--time before the process gets a -9 --></span>
+<span class="nt"><property></span>
+ <span class="nt"><name></span>yarn.nodemanager.sleep-delay-before-sigkill.ms<span class="nt"></name></span>
+ <span class="nt"><value></span>30000<span class="nt"></value></span>
+<span class="nt"></property></span>
+</pre></div>
</div>
<div id="footer">
Modified: incubator/slider/site/trunk/content/developing/building.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/developing/building.md?rev=1608633&r1=1608632&r2=1608633&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/developing/building.md (original)
+++ incubator/slider/site/trunk/content/developing/building.md Tue Jul 8 00:49:34 2014
@@ -46,11 +46,28 @@ You will need a version of Maven 3.0+, s
You need a copy of the `protoc` compiler for protobuf compilation
1. OS/X: `brew install protobuf`
-1. Others: consult (Building Hadoop documentation)[http://wiki.apache.org/hadoop/HowToContribute].
+1. Others: consult [Building Hadoop documentation](http://wiki.apache.org/hadoop/HowToContribute).
The version of `protoc` installed must be the same as that used by Hadoop itself.
This is absolutely critical to prevent JAR version problems.
+### Build, install and run unit tests
+
+ mvn clean install
+
+Slider code base includes unit tests and functional tests. By default functional tests are not run as these tests are developed to be run against live hadoop clusters and require some manual setup. You can run them based as described at [functional test](/developing/functional_tests.html).
+
+
+### Create Slider Package
+
+ mvn clean site:site site:stage package -DskipTests
+
+
+***At this point you are ready to use the Slider toolset. The build instructions below are optional and are proven to be useful when debugging deep into the hadoop code base.***
+
+----------------
+
+
## Building a compatible Hadoop version
@@ -106,7 +123,7 @@ This creates an expanded version of Hado
from this directory. Do note that unless you have the native code built for
your target platform, Hadoop will be slower.
-## building a compatible HBase version
+## Building a compatible HBase version
If you need to build a version of HBase -rather than use a released version,
here are the instructions (for the hbase-0.98 release branch)