You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@slider.apache.org by sm...@apache.org on 2014/07/08 02:49:35 UTC
svn commit: r1608633 [3/7] - in /incubator/slider/site: content/design/
content/design/registry/ content/design/specification/ content/developing/
content/docs/ content/docs/configuration/ content/docs/slider_specs/
trunk/content/developing/
Modified: incubator/slider/site/content/design/specification/slider-model.html
URL: http://svn.apache.org/viewvc/incubator/slider/site/content/design/specification/slider-model.html?rev=1608633&r1=1608632&r2=1608633&view=diff
==============================================================================
--- incubator/slider/site/content/design/specification/slider-model.html (original)
+++ incubator/slider/site/content/design/specification/slider-model.html Tue Jul 8 00:49:34 2014
@@ -189,69 +189,81 @@ Two key references are</p>
<h2 id="yarn">YARN</h2>
<p>From the perspective of YARN application, The YARN runtime is a state, <code>YARN</code>,
comprised of: <code>(Apps, Queues, Nodes)</code></p>
-<pre class="codehilite"><code>Apps: Map[AppId, ApplicationReport]</code></pre>
+<div class="codehilite"><pre><span class="n">Apps</span><span class="o">:</span> <span class="n">Map</span><span class="o">[</span><span class="n">AppId</span><span class="o">,</span> <span class="n">ApplicationReport</span><span class="o">]</span>
+</pre></div>
<p>An application has a name, an application report and a list of outstanding requests</p>
-<pre class="codehilite"><code>App: (Name, report: ApplicationReport, Requests:List[AmRequest])</code></pre>
+<div class="codehilite"><pre><span class="nl">App:</span> <span class="p">(</span><span class="n">Name</span><span class="p">,</span> <span class="n">report</span><span class="o">:</span> <span class="n">ApplicationReport</span><span class="p">,</span> <span class="n">Requests</span><span class="o">:</span><span class="n">List</span><span class="p">[</span><span class="n">AmRequest</span><span class="p">])</span>
+</pre></div>
<p>An application report contains a mixture of static and dynamic state of the application
and the AM.</p>
-<pre class="codehilite"><code>ApplicationReport: AppId, Type, User, YarnApplicationState, AmContainer, RpcPort, TrackingURL,</code></pre>
+<div class="codehilite"><pre><span class="n">ApplicationReport</span><span class="o">:</span> <span class="n">AppId</span><span class="o">,</span> <span class="n">Type</span><span class="o">,</span> <span class="n">User</span><span class="o">,</span> <span class="n">YarnApplicationState</span><span class="o">,</span> <span class="n">AmContainer</span><span class="o">,</span> <span class="n">RpcPort</span><span class="o">,</span> <span class="n">TrackingURL</span><span class="o">,</span>
+</pre></div>
<p>YARN applications have a number of states. These are ordered such that if the
<code>state.ordinal() > RUNNING.ordinal()</code> then the application has entered an exit state.</p>
-<pre class="codehilite"><code>YarnApplicationState : [NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED ]</code></pre>
+<div class="codehilite"><pre><span class="n">YarnApplicationState</span> <span class="o">:</span> <span class="p">[</span><span class="n">NEW</span><span class="p">,</span> <span class="n">NEW_SAVING</span><span class="p">,</span> <span class="n">SUBMITTED</span><span class="p">,</span> <span class="n">ACCEPTED</span><span class="p">,</span> <span class="n">RUNNING</span><span class="p">,</span> <span class="n">FINISHED</span><span class="p">,</span> <span class="n">FAILED</span><span class="p">,</span> <span class="n">KILLED</span> <span class="p">]</span>
+</pre></div>
<p>AMs can request containers to be added or released </p>
-<pre class="codehilite"><code>AmRequest = { add-container(priority, requirements), release(containerId)}</code></pre>
+<div class="codehilite"><pre><span class="n">AmRequest</span> <span class="o">=</span> <span class="p">{</span> <span class="n">add</span><span class="o">-</span><span class="n">container</span><span class="p">(</span><span class="n">priority</span><span class="p">,</span> <span class="n">requirements</span><span class="p">),</span> <span class="n">release</span><span class="p">(</span><span class="n">containerId</span><span class="p">)}</span>
+</pre></div>
<p>Job queues are named queues of job requests; there is always a queue called <code>"default"</code></p>
-<pre class="codehilite"><code>Queues: Map[String:Queue]
- Queue: List[Requests]
- Request = {
- launch(app-name, app-type, requirements, context)
- }
- Context: (localized-resources: Map[String,URL], command)</code></pre>
+<div class="codehilite"><pre><span class="n">Queues</span><span class="o">:</span> <span class="n">Map</span><span class="o">[</span><span class="n">String</span><span class="o">:</span><span class="n">Queue</span><span class="o">]</span>
+ <span class="n">Queue</span><span class="o">:</span> <span class="n">List</span><span class="o">[</span><span class="n">Requests</span><span class="o">]</span>
+ <span class="n">Request</span> <span class="o">=</span> <span class="o">{</span>
+ <span class="n">launch</span><span class="o">(</span><span class="n">app</span><span class="o">-</span><span class="n">name</span><span class="o">,</span> <span class="n">app</span><span class="o">-</span><span class="n">type</span><span class="o">,</span> <span class="n">requirements</span><span class="o">,</span> <span class="n">context</span><span class="o">)</span>
+ <span class="o">}</span>
+ <span class="n">Context</span><span class="o">:</span> <span class="o">(</span><span class="n">localized</span><span class="o">-</span><span class="n">resources</span><span class="o">:</span> <span class="n">Map</span><span class="o">[</span><span class="n">String</span><span class="o">,</span><span class="n">URL</span><span class="o">],</span> <span class="n">command</span><span class="o">)</span>
+</pre></div>
<p>This is doesn't completely model the cluster from the AM perspective -there's no
notion of node operations (launching code in a container) or events coming from YARN.</p>
<p>The <code>Nodes</code> structure models the nodes in a cluster</p>
-<pre class="codehilite"><code>Nodes: Map[nodeID,(name, containers:List[Container])]</code></pre>
+<div class="codehilite"><pre><span class="n">Nodes</span><span class="o">:</span> <span class="n">Map</span><span class="o">[</span><span class="n">nodeID</span><span class="o">,(</span><span class="n">name</span><span class="o">,</span> <span class="n">containers</span><span class="o">:</span><span class="n">List</span><span class="o">[</span><span class="n">Container</span><span class="o">])]</span>
+</pre></div>
<p>A container contains some state</p>
-<pre class="codehilite"><code>Container: (containerId, appId, context)</code></pre>
+<div class="codehilite"><pre><span class="nl">Container:</span> <span class="p">(</span><span class="n">containerId</span><span class="p">,</span> <span class="n">appId</span><span class="p">,</span> <span class="n">context</span><span class="p">)</span>
+</pre></div>
<p>The containers in a cluster are the aggregate set of all containers across
all nodes</p>
-<pre class="codehilite"><code>def containers(YARN) =
- [c for n in keys(YARN.Nodes) for c in YARN.Nodes[n].Containers ]</code></pre>
+<div class="codehilite"><pre><span class="n">def</span> <span class="n">containers</span><span class="p">(</span><span class="n">YARN</span><span class="p">)</span> <span class="o">=</span>
+ <span class="p">[</span><span class="n">c</span> <span class="k">for</span> <span class="n">n</span> <span class="n">in</span> <span class="n">keys</span><span class="p">(</span><span class="n">YARN</span><span class="p">.</span><span class="n">Nodes</span><span class="p">)</span> <span class="k">for</span> <span class="n">c</span> <span class="n">in</span> <span class="n">YARN</span><span class="p">.</span><span class="n">Nodes</span><span class="p">[</span><span class="n">n</span><span class="p">].</span><span class="n">Containers</span> <span class="p">]</span>
+</pre></div>
<p>The containers of an application are all containers that are considered owned by it,</p>
-<pre class="codehilite"><code>def app-containers(YARN, appId: AppId) =
- [c in containers(YARN) where c.appId == appId ]</code></pre>
+<div class="codehilite"><pre><span class="n">def</span> <span class="n">app</span><span class="o">-</span><span class="n">containers</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">appId</span><span class="o">:</span> <span class="n">AppId</span><span class="p">)</span> <span class="o">=</span>
+ <span class="p">[</span><span class="n">c</span> <span class="n">in</span> <span class="n">containers</span><span class="p">(</span><span class="n">YARN</span><span class="p">)</span> <span class="n">where</span> <span class="n">c</span><span class="p">.</span><span class="n">appId</span> <span class="o">==</span> <span class="n">appId</span> <span class="p">]</span>
+</pre></div>
<h3 id="operations-predicates-used-the-specifications">Operations & predicates used the specifications</h3>
-<pre class="codehilite"><code>def applications(YARN, type) =
- [ app.report for app in YARN.Apps.values where app.report.Type == type]
+<div class="codehilite"><pre><span class="n">def</span> <span class="n">applications</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">type</span><span class="p">)</span> <span class="o">=</span>
+ <span class="p">[</span> <span class="n">app</span><span class="p">.</span><span class="n">report</span> <span class="k">for</span> <span class="n">app</span> <span class="n">in</span> <span class="n">YARN</span><span class="p">.</span><span class="n">Apps</span><span class="p">.</span><span class="n">values</span> <span class="n">where</span> <span class="n">app</span><span class="p">.</span><span class="n">report</span><span class="p">.</span><span class="n">Type</span> <span class="o">==</span> <span class="n">type</span><span class="p">]</span>
-def user-applications(YARN, type, user)
- [a in applications(YARN, type) where: a.User == user]</code></pre>
+<span class="n">def</span> <span class="n">user</span><span class="o">-</span><span class="n">applications</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">type</span><span class="p">,</span> <span class="n">user</span><span class="p">)</span>
+ <span class="p">[</span><span class="n">a</span> <span class="n">in</span> <span class="n">applications</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">type</span><span class="p">)</span> <span class="n">where</span><span class="o">:</span> <span class="n">a</span><span class="p">.</span><span class="n">User</span> <span class="o">==</span> <span class="n">user</span><span class="p">]</span>
+</pre></div>
<h2 id="usergroupinformation">UserGroupInformation</h2>
<p>Applications are launched and executed on hosts computers: either client machines
or nodes in the cluster, these have their own state which may need modeling</p>
-<pre class="codehilite"><code>HostState: Map[String, String]</code></pre>
+<div class="codehilite"><pre><span class="n">HostState</span><span class="o">:</span> <span class="n">Map</span><span class="o">[</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">]</span>
+</pre></div>
<p>A key part of the host state is actually the identity of the current user,
@@ -266,77 +278,84 @@ If it did they would be used throughout
<h2 id="slider-model">Slider Model</h2>
<h3 id="cluster-name">Cluster name</h3>
<p>A valid cluster name is a name of length > 1 which follows the internet hostname scheme of letter followed by letter or digit</p>
-<pre class="codehilite"><code>def valid-cluster-name(c) =
- len(c)> 0
- and c[0] in ['a'..'z']
- and c[1] in (['a'..'z'] + ['-'] + ['0..9'])</code></pre>
+<div class="codehilite"><pre><span class="n">def</span> <span class="n">valid</span><span class="o">-</span><span class="n">cluster</span><span class="o">-</span><span class="n">name</span><span class="p">(</span><span class="n">c</span><span class="p">)</span> <span class="o">=</span>
+ <span class="n">len</span><span class="p">(</span><span class="n">c</span><span class="p">)</span><span class="o">></span> <span class="mi">0</span>
+ <span class="n">and</span> <span class="n">c</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="n">in</span> <span class="p">[</span><span class="sc">'a'</span><span class="p">..</span><span class="sc">'z'</span><span class="p">]</span>
+ <span class="n">and</span> <span class="n">c</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="n">in</span> <span class="p">([</span><span class="sc">'a'</span><span class="p">..</span><span class="sc">'z'</span><span class="p">]</span> <span class="o">+</span> <span class="p">[</span><span class="sc">'-'</span><span class="p">]</span> <span class="o">+</span> <span class="p">[</span><span class="err">'</span><span class="mf">0..9</span><span class="err">'</span><span class="p">])</span>
+</pre></div>
<h3 id="persistent-cluster-state">Persistent Cluster State</h3>
<p>A Slider cluster's persistent state is stored in a path</p>
-<pre class="codehilite"><code>def cluster-path(FS, clustername) = user-home(FS) + ["clusters", clustername]
-def cluster-json-path(FS, clustername) = cluster-path(FS, clustername) + ["cluster.json"]
-def original-conf-path(FS, clustername) = cluster-path(FS, clustername) + ["original"]
-def generated-conf-path(FS, clustername) = cluster-path(FS, clustername) + ["generated"]
-def data-path(FS, clustername) = cluster-path(FS, clustername) + ["data"]</code></pre>
+<div class="codehilite"><pre><span class="n">def</span> <span class="n">cluster</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">clustername</span><span class="p">)</span> <span class="o">=</span> <span class="n">user</span><span class="o">-</span><span class="n">home</span><span class="p">(</span><span class="n">FS</span><span class="p">)</span> <span class="o">+</span> <span class="p">[</span><span class="s">"clusters"</span><span class="p">,</span> <span class="n">clustername</span><span class="p">]</span>
+<span class="n">def</span> <span class="n">cluster</span><span class="o">-</span><span class="n">json</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">clustername</span><span class="p">)</span> <span class="o">=</span> <span class="n">cluster</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">clustername</span><span class="p">)</span> <span class="o">+</span> <span class="p">[</span><span class="s">"cluster.json"</span><span class="p">]</span>
+<span class="n">def</span> <span class="n">original</span><span class="o">-</span><span class="n">conf</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">clustername</span><span class="p">)</span> <span class="o">=</span> <span class="n">cluster</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">clustername</span><span class="p">)</span> <span class="o">+</span> <span class="p">[</span><span class="s">"original"</span><span class="p">]</span>
+<span class="n">def</span> <span class="n">generated</span><span class="o">-</span><span class="n">conf</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">clustername</span><span class="p">)</span> <span class="o">=</span> <span class="n">cluster</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">clustername</span><span class="p">)</span> <span class="o">+</span> <span class="p">[</span><span class="s">"generated"</span><span class="p">]</span>
+<span class="n">def</span> <span class="n">data</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">clustername</span><span class="p">)</span> <span class="o">=</span> <span class="n">cluster</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">clustername</span><span class="p">)</span> <span class="o">+</span> <span class="p">[</span><span class="s">"data"</span><span class="p">]</span>
+</pre></div>
<p>When a cluster is built/created the specified original configuration directory
is copied to <code>original-conf-path(FS, clustername)</code>; this is patched for the
specific instance bindings and saved into <code>generated-conf-path(FS, clustername)</code>.</p>
<p>A cluster <em>exists</em> if all of these paths are found:</p>
-<pre class="codehilite"><code>def cluster-exists(FS, clustername) =
- is-dir(FS, cluster-path(FS, clustername))
- and is-file(FS, cluster-json-path(FS, clustername))
- and is-dir(FS, original-conf-path(FS, clustername))
- and generated-conf-path(FS, original-conf-path(FS, clustername))</code></pre>
+<div class="codehilite"><pre><span class="n">def</span> <span class="n">cluster</span><span class="o">-</span><span class="n">exists</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">clustername</span><span class="p">)</span> <span class="o">=</span>
+ <span class="n">is</span><span class="o">-</span><span class="n">dir</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">cluster</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">clustername</span><span class="p">))</span>
+ <span class="n">and</span> <span class="n">is</span><span class="o">-</span><span class="n">file</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">cluster</span><span class="o">-</span><span class="n">json</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">clustername</span><span class="p">))</span>
+ <span class="n">and</span> <span class="n">is</span><span class="o">-</span><span class="n">dir</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">original</span><span class="o">-</span><span class="n">conf</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">clustername</span><span class="p">))</span>
+ <span class="n">and</span> <span class="n">generated</span><span class="o">-</span><span class="n">conf</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">original</span><span class="o">-</span><span class="n">conf</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">clustername</span><span class="p">))</span>
+</pre></div>
<p>A cluster is considered <code>running</code> if there is a Slider application type belonging to the current user in one of the states
<code>{NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING}</code>. </p>
-<pre class="codehilite"><code>def final-yarn-states = {FINISHED, FAILED, KILLED }
+<div class="codehilite"><pre><span class="n">def</span> <span class="n">final</span><span class="o">-</span><span class="n">yarn</span><span class="o">-</span><span class="n">states</span> <span class="o">=</span> <span class="p">{</span><span class="n">FINISHED</span><span class="p">,</span> <span class="n">FAILED</span><span class="p">,</span> <span class="n">KILLED</span> <span class="p">}</span>
-def slider-app-instances(YARN, clustername, user) =
- [a in user-applications(YARN, "slider", user) where:
- and a.Name == clustername]
-
-def slider-app-running-instances(YARN, clustername, user) =
- [a in slider-app-instances(YARN, user, clustername) where:
- not a.YarnApplicationState in final-yarn-state]
-
-def slider-app-running(YARN, clustername, user) =
- [] != slider-app-running-instances(YARN, clustername, user)
-
-def slider-app-live-instances(YARN, clustername, user) =
- [a in slider-app-instances(YARN, user, clustername) where:
- a.YarnApplicationState == RUNNING]
-
-def slider-app-live(YARN, clustername, user) =
- [] != slider-app-live-instances(YARN, clustername, user)</code></pre>
+<span class="n">def</span> <span class="n">slider</span><span class="o">-</span><span class="n">app</span><span class="o">-</span><span class="n">instances</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">clustername</span><span class="p">,</span> <span class="n">user</span><span class="p">)</span> <span class="o">=</span>
+ <span class="p">[</span><span class="n">a</span> <span class="n">in</span> <span class="n">user</span><span class="o">-</span><span class="n">applications</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="s">"slider"</span><span class="p">,</span> <span class="n">user</span><span class="p">)</span> <span class="n">where</span><span class="o">:</span>
+ <span class="n">and</span> <span class="n">a</span><span class="p">.</span><span class="n">Name</span> <span class="o">==</span> <span class="n">clustername</span><span class="p">]</span>
+
+<span class="n">def</span> <span class="n">slider</span><span class="o">-</span><span class="n">app</span><span class="o">-</span><span class="n">running</span><span class="o">-</span><span class="n">instances</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">clustername</span><span class="p">,</span> <span class="n">user</span><span class="p">)</span> <span class="o">=</span>
+ <span class="p">[</span><span class="n">a</span> <span class="n">in</span> <span class="n">slider</span><span class="o">-</span><span class="n">app</span><span class="o">-</span><span class="n">instances</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">user</span><span class="p">,</span> <span class="n">clustername</span><span class="p">)</span> <span class="n">where</span><span class="o">:</span>
+ <span class="n">not</span> <span class="n">a</span><span class="p">.</span><span class="n">YarnApplicationState</span> <span class="n">in</span> <span class="n">final</span><span class="o">-</span><span class="n">yarn</span><span class="o">-</span><span class="n">state</span><span class="p">]</span>
+
+<span class="n">def</span> <span class="n">slider</span><span class="o">-</span><span class="n">app</span><span class="o">-</span><span class="n">running</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">clustername</span><span class="p">,</span> <span class="n">user</span><span class="p">)</span> <span class="o">=</span>
+ <span class="p">[]</span> <span class="o">!=</span> <span class="n">slider</span><span class="o">-</span><span class="n">app</span><span class="o">-</span><span class="n">running</span><span class="o">-</span><span class="n">instances</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">clustername</span><span class="p">,</span> <span class="n">user</span><span class="p">)</span>
+
+<span class="n">def</span> <span class="n">slider</span><span class="o">-</span><span class="n">app</span><span class="o">-</span><span class="n">live</span><span class="o">-</span><span class="n">instances</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">clustername</span><span class="p">,</span> <span class="n">user</span><span class="p">)</span> <span class="o">=</span>
+ <span class="p">[</span><span class="n">a</span> <span class="n">in</span> <span class="n">slider</span><span class="o">-</span><span class="n">app</span><span class="o">-</span><span class="n">instances</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">user</span><span class="p">,</span> <span class="n">clustername</span><span class="p">)</span> <span class="n">where</span><span class="o">:</span>
+ <span class="n">a</span><span class="p">.</span><span class="n">YarnApplicationState</span> <span class="o">==</span> <span class="n">RUNNING</span><span class="p">]</span>
+
+<span class="n">def</span> <span class="n">slider</span><span class="o">-</span><span class="n">app</span><span class="o">-</span><span class="n">live</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">clustername</span><span class="p">,</span> <span class="n">user</span><span class="p">)</span> <span class="o">=</span>
+ <span class="p">[]</span> <span class="o">!=</span> <span class="n">slider</span><span class="o">-</span><span class="n">app</span><span class="o">-</span><span class="n">live</span><span class="o">-</span><span class="n">instances</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">clustername</span><span class="p">,</span> <span class="n">user</span><span class="p">)</span>
+</pre></div>
<h3 id="invariant-there-must-never-be-more-than-one-running-instance-of-a-named-slider-cluster">Invariant: there must never be more than one running instance of a named Slider cluster</h3>
<p>There must never be more than one instance of the same Slider cluster running:</p>
-<pre class="codehilite"><code>forall a in user-applications(YARN, "slider", user):
- len(slider-app-running-instances(YARN, a.Name, user)) <= 1</code></pre>
+<div class="codehilite"><pre><span class="n">forall</span> <span class="n">a</span> <span class="n">in</span> <span class="n">user</span><span class="o">-</span><span class="n">applications</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="s">"slider"</span><span class="p">,</span> <span class="n">user</span><span class="p">)</span><span class="o">:</span>
+ <span class="n">len</span><span class="p">(</span><span class="n">slider</span><span class="o">-</span><span class="n">app</span><span class="o">-</span><span class="n">running</span><span class="o">-</span><span class="n">instances</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">a</span><span class="p">.</span><span class="n">Name</span><span class="p">,</span> <span class="n">user</span><span class="p">))</span> <span class="o"><=</span> <span class="mi">1</span>
+</pre></div>
<p>There may be multiple instances in a finished state, and one running instance alongside multiple finished instances -the applications
that work with Slider MUST select a running cluster ahead of any terminated clusters.</p>
<h3 id="containers-of-an-application">Containers of an application</h3>
<p>The containers of a slider application are the set of containers of that application</p>
-<pre class="codehilite"><code>def slider-app-containers(YARN, clustername, user) =
- app-containers(YARN, appid where
- appid = slider-app-running-instances(YARN, clustername, user)[0])</code></pre>
+<div class="codehilite"><pre><span class="n">def</span> <span class="n">slider</span><span class="o">-</span><span class="n">app</span><span class="o">-</span><span class="n">containers</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">clustername</span><span class="p">,</span> <span class="n">user</span><span class="p">)</span> <span class="o">=</span>
+ <span class="n">app</span><span class="o">-</span><span class="n">containers</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">appid</span> <span class="n">where</span>
+ <span class="n">appid</span> <span class="o">=</span> <span class="n">slider</span><span class="o">-</span><span class="n">app</span><span class="o">-</span><span class="n">running</span><span class="o">-</span><span class="n">instances</span><span class="p">(</span><span class="n">YARN</span><span class="p">,</span> <span class="n">clustername</span><span class="p">,</span> <span class="n">user</span><span class="p">)[</span><span class="mi">0</span><span class="p">])</span>
+</pre></div>
<h3 id="rpc-access-to-a-slider-cluster">RPC Access to a slider cluster</h3>
<p>An application is accepting RPC requests for a given protocol if there is a port binding
defined and it is possible to authenticate a connection using the specified protocol</p>
-<pre class="codehilite"><code> def rpc-connection(appReport, protocol) =
- appReport.host != null
- appReport.rpcPort != 0
- and RPC.getProtocolProxy(appReport.host, appReport.rpcPort, protocol)</code></pre>
+<div class="codehilite"><pre> <span class="n">def</span> <span class="n">rpc</span><span class="o">-</span><span class="n">connection</span><span class="p">(</span><span class="n">appReport</span><span class="p">,</span> <span class="n">protocol</span><span class="p">)</span> <span class="o">=</span>
+ <span class="n">appReport</span><span class="p">.</span><span class="n">host</span> <span class="o">!=</span> <span class="n">null</span>
+ <span class="n">appReport</span><span class="p">.</span><span class="n">rpcPort</span> <span class="o">!=</span> <span class="mi">0</span>
+ <span class="n">and</span> <span class="n">RPC</span><span class="p">.</span><span class="n">getProtocolProxy</span><span class="p">(</span><span class="n">appReport</span><span class="p">.</span><span class="n">host</span><span class="p">,</span> <span class="n">appReport</span><span class="p">.</span><span class="n">rpcPort</span><span class="p">,</span> <span class="n">protocol</span><span class="p">)</span>
+</pre></div>
<p>Being able to open an RPC port is the strongest definition of liveness possible
@@ -369,9 +388,10 @@ is well-defined if</p>
<li>The entry <code>name</code> must match a supported provider</li>
<li>
<p>Any elements that name the cluster match the cluster name as defined by the path to the cluster:</p>
-<pre class="codehilite"><code>originConfigurationPath == original-conf-path(FS, clustername)
-generatedConfigurationPath == generated-conf-path(FS, clustername)
-dataPath == data-path(FS, clustername)</code></pre>
+<div class="codehilite"><pre><span class="n">originConfigurationPath</span> <span class="o">==</span> <span class="n">original</span><span class="o">-</span><span class="n">conf</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">clustername</span><span class="p">)</span>
+<span class="n">generatedConfigurationPath</span> <span class="o">==</span> <span class="n">generated</span><span class="o">-</span><span class="n">conf</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">clustername</span><span class="p">)</span>
+<span class="n">dataPath</span> <span class="o">==</span> <span class="n">data</span><span class="o">-</span><span class="n">path</span><span class="p">(</span><span class="n">FS</span><span class="p">,</span> <span class="n">clustername</span><span class="p">)</span>
+</pre></div>
</li>
@@ -383,16 +403,18 @@ defined by (<code>options/zookeeper.host
<li>If <code>options/cluster.application.image.path</code> is defined, it must exist and be readable by the user.</li>
<li>
<p>It must declare a type that maps to a provider entry in the Slider client's XML configuration:</p>
-<pre class="codehilite"><code>len(clusterspec["type"]) > 0
-clientconfig["slider.provider."+ clusterspec["type"]] != null</code></pre>
+<div class="codehilite"><pre><span class="n">len</span><span class="p">(</span><span class="n">clusterspec</span><span class="p">[</span><span class="s">"type"</span><span class="p">])</span> <span class="o">></span> <span class="mi">0</span>
+<span class="n">clientconfig</span><span class="p">[</span><span class="s">"slider.provider."</span><span class="o">+</span> <span class="n">clusterspec</span><span class="p">[</span><span class="s">"type"</span><span class="p">]]</span> <span class="o">!=</span> <span class="n">null</span>
+</pre></div>
</li>
<li>
<p>That entry must map to a class on the classpath which can be instantiated
and cast to <code>SliderProviderFactory</code>.</p>
-<pre class="codehilite"><code>let classname = clientconfig["slider.provider."+ clusterspec["type"]]
-(Class.forName(classname).newInstance()) instanceof SliderProviderFactory</code></pre>
+<div class="codehilite"><pre><span class="n">let</span> <span class="n">classname</span> <span class="o">=</span> <span class="n">clientconfig</span><span class="p">[</span><span class="s">"slider.provider."</span><span class="o">+</span> <span class="n">clusterspec</span><span class="p">[</span><span class="s">"type"</span><span class="p">]]</span>
+<span class="p">(</span><span class="n">Class</span><span class="p">.</span><span class="n">forName</span><span class="p">(</span><span class="n">classname</span><span class="p">).</span><span class="n">newInstance</span><span class="p">())</span> <span class="n">instanceof</span> <span class="n">SliderProviderFactory</span>
+</pre></div>
</li>
Modified: incubator/slider/site/content/developing/building.html
URL: http://svn.apache.org/viewvc/incubator/slider/site/content/developing/building.html?rev=1608633&r1=1608632&r2=1608633&view=diff
==============================================================================
--- incubator/slider/site/content/developing/building.html (original)
+++ incubator/slider/site/content/developing/building.html Tue Jul 8 00:49:34 2014
@@ -180,7 +180,8 @@ of the local host working. Tests will fa
<p>Slider is built on Java 7 -please have a JDK for Java 7 or 8 set up</p>
<h3 id="maven">Maven</h3>
<p>You will need a version of Maven 3.0+, set up with enough memory</p>
-<pre class="codehilite"><code>MAVEN_OPTS=-Xms256m -Xmx512m -Djava.awt.headless=true</code></pre>
+<div class="codehilite"><pre><span class="n">MAVEN_OPTS</span><span class="o">=-</span><span class="n">Xms256m</span> <span class="o">-</span><span class="n">Xmx512m</span> <span class="o">-</span><span class="n">Djava</span><span class="p">.</span><span class="n">awt</span><span class="p">.</span><span class="n">headless</span><span class="o">=</span><span class="nb">true</span>
+</pre></div>
<p><em>Important</em>: As of October 6, 2013, Maven 3.1 is not supported due to
@@ -189,10 +190,23 @@ of the local host working. Tests will fa
<p>You need a copy of the <code>protoc</code> compiler for protobuf compilation</p>
<ol>
<li>OS/X: <code>brew install protobuf</code></li>
-<li>Others: consult (Building Hadoop documentation)[http://wiki.apache.org/hadoop/HowToContribute].</li>
+<li>Others: consult <a href="http://wiki.apache.org/hadoop/HowToContribute">Building Hadoop documentation</a>.</li>
</ol>
<p>The version of <code>protoc</code> installed must be the same as that used by Hadoop itself.
This is absolutely critical to prevent JAR version problems.</p>
+<h3 id="build-install-and-run-unit-tests">Build, install and run unit tests</h3>
+<div class="codehilite"><pre> <span class="n">mvn</span> <span class="n">clean</span> <span class="n">install</span>
+</pre></div>
+
+
+<p>Slider code base includes unit tests and functional tests. By default functional tests are not run as these tests are developed to be run against live hadoop clusters and require some manual setup. You can run them based as described at <a href="/developing/functional_tests.html">functional test</a>. </p>
+<h3 id="create-slider-package">Create Slider Package</h3>
+<div class="codehilite"><pre> <span class="n">mvn</span> <span class="n">clean</span> <span class="n">site</span><span class="o">:</span><span class="n">site</span> <span class="n">site</span><span class="o">:</span><span class="n">stage</span> <span class="n">package</span> <span class="o">-</span><span class="n">DskipTests</span>
+</pre></div>
+
+
+<p><strong><em>At this point you are ready to use the Slider toolset. The build instructions below are optional and are proven to be useful when debugging deep into the hadoop code base.</em></strong></p>
+<hr />
<h2 id="building-a-compatible-hadoop-version">Building a compatible Hadoop version</h2>
<p>Slider is built against Hadoop 2 -you can download and install
a copy from the <a href="http://hadoop.apache.org">Apache Hadoop Web Site</a>.</p>
@@ -201,86 +215,99 @@ to have a local version of Hadoop -so th
Hadoop as well in Slider.</p>
<p>To build and install locally, check out apache svn/github, branch <code>release-2.4.0</code>,
and create a branch off that tag</p>
-<pre class="codehilite"><code>git clone git://git.apache.org/hadoop-common.git
-cd hadoop-common
-git remote rename origin apache
-git fetch --tags apache
-git checkout release-2.4.0 --
-git checkout -b release-2.4.0</code></pre>
+<div class="codehilite"><pre><span class="n">git</span> <span class="n">clone</span> <span class="n">git</span><span class="o">:</span><span class="c1">//git.apache.org/hadoop-common.git </span>
+<span class="n">cd</span> <span class="n">hadoop</span><span class="o">-</span><span class="n">common</span>
+<span class="n">git</span> <span class="n">remote</span> <span class="n">rename</span> <span class="n">origin</span> <span class="n">apache</span>
+<span class="n">git</span> <span class="n">fetch</span> <span class="o">--</span><span class="n">tags</span> <span class="n">apache</span>
+<span class="n">git</span> <span class="n">checkout</span> <span class="n">release</span><span class="o">-</span><span class="mf">2.4.0</span> <span class="o">--</span>
+<span class="n">git</span> <span class="n">checkout</span> <span class="o">-</span><span class="n">b</span> <span class="n">release</span><span class="o">-</span><span class="mf">2.4.0</span>
+</pre></div>
<p>For the scripts below, set the <code>HADOOP_VERSION</code> variable to the version</p>
-<pre class="codehilite"><code>export HADOOP_VERSION=2.4.0</code></pre>
+<div class="codehilite"><pre><span class="n">export</span> <span class="n">HADOOP_VERSION</span><span class="o">=</span><span class="mf">2.4.0</span>
+</pre></div>
<p>or, for building against a pre-release version of Hadoop 2.4</p>
-<pre class="codehilite"><code>git checkout branch-2
-export HADOOP_VERSION=2.4.0-SNAPSHOT</code></pre>
+<div class="codehilite"><pre><span class="n">git</span> <span class="n">checkout</span> <span class="n">branch</span><span class="o">-</span><span class="mi">2</span>
+<span class="n">export</span> <span class="n">HADOOP_VERSION</span><span class="o">=</span><span class="mf">2.4.0</span><span class="o">-</span><span class="n">SNAPSHOT</span>
+</pre></div>
<p>To build and install it locally, skipping the tests:</p>
-<pre class="codehilite"><code>mvn clean install -DskipTests</code></pre>
+<div class="codehilite"><pre><span class="n">mvn</span> <span class="n">clean</span> <span class="n">install</span> <span class="o">-</span><span class="n">DskipTests</span>
+</pre></div>
<p>To make a tarball for use in test runs:</p>
-<pre class="codehilite"><code>#On osx
-mvn clean install package -Pdist -Dtar -DskipTests -Dmaven.javadoc.skip=true
+<div class="codehilite"><pre><span class="cp">#On osx</span>
+<span class="n">mvn</span> <span class="n">clean</span> <span class="n">install</span> <span class="n">package</span> <span class="o">-</span><span class="n">Pdist</span> <span class="o">-</span><span class="n">Dtar</span> <span class="o">-</span><span class="n">DskipTests</span> <span class="o">-</span><span class="n">Dmaven</span><span class="p">.</span><span class="n">javadoc</span><span class="p">.</span><span class="n">skip</span><span class="o">=</span><span class="nb">true</span>
-# on linux
-mvn clean package -Pdist -Pnative -Dtar -DskipTests -Dmaven.javadoc.skip=true</code></pre>
+<span class="cp"># on linux</span>
+<span class="n">mvn</span> <span class="n">clean</span> <span class="n">package</span> <span class="o">-</span><span class="n">Pdist</span> <span class="o">-</span><span class="n">Pnative</span> <span class="o">-</span><span class="n">Dtar</span> <span class="o">-</span><span class="n">DskipTests</span> <span class="o">-</span><span class="n">Dmaven</span><span class="p">.</span><span class="n">javadoc</span><span class="p">.</span><span class="n">skip</span><span class="o">=</span><span class="nb">true</span>
+</pre></div>
<p>Then expand this</p>
-<pre class="codehilite"><code>pushd hadoop-dist/target/
-gunzip hadoop-$HADOOP_VERSION.tar.gz
-tar -xvf hadoop-$HADOOP_VERSION.tar
-popd</code></pre>
+<div class="codehilite"><pre><span class="n">pushd</span> <span class="n">hadoop</span><span class="o">-</span><span class="n">dist</span><span class="o">/</span><span class="n">target</span><span class="o">/</span>
+<span class="n">gunzip</span> <span class="n">hadoop</span><span class="o">-</span><span class="err">$</span><span class="n">HADOOP_VERSION</span><span class="p">.</span><span class="n">tar</span><span class="p">.</span><span class="n">gz</span>
+<span class="n">tar</span> <span class="o">-</span><span class="n">xvf</span> <span class="n">hadoop</span><span class="o">-</span><span class="err">$</span><span class="n">HADOOP_VERSION</span><span class="p">.</span><span class="n">tar</span>
+<span class="n">popd</span>
+</pre></div>
<p>This creates an expanded version of Hadoop. You can now actually run Hadoop
from this directory. Do note that unless you have the native code built for
your target platform, Hadoop will be slower. </p>
-<h2 id="building-a-compatible-hbase-version">building a compatible HBase version</h2>
+<h2 id="building-a-compatible-hbase-version">Building a compatible HBase version</h2>
<p>If you need to build a version of HBase -rather than use a released version,
here are the instructions (for the hbase-0.98 release branch)</p>
<p>Checkout the HBase <code>trunk</code> branch from apache svn/github. </p>
-<pre class="codehilite"><code>git clone git://git.apache.org/hbase.git
-cd hbase
-git remote rename origin apache
-git fetch --tags apache</code></pre>
+<div class="codehilite"><pre><span class="n">git</span> <span class="n">clone</span> <span class="n">git</span><span class="o">:</span><span class="c1">//git.apache.org/hbase.git</span>
+<span class="n">cd</span> <span class="n">hbase</span>
+<span class="n">git</span> <span class="n">remote</span> <span class="n">rename</span> <span class="n">origin</span> <span class="n">apache</span>
+<span class="n">git</span> <span class="n">fetch</span> <span class="o">--</span><span class="n">tags</span> <span class="n">apache</span>
+</pre></div>
<p>then</p>
-<pre class="codehilite"><code>git checkout -b apache/0.98</code></pre>
+<div class="codehilite"><pre><span class="n">git</span> <span class="n">checkout</span> <span class="o">-</span><span class="n">b</span> <span class="n">apache</span><span class="o">/</span><span class="mf">0.98</span>
+</pre></div>
<p>or</p>
-<pre class="codehilite"><code>git checkout tags/0.98.1</code></pre>
+<div class="codehilite"><pre><span class="n">git</span> <span class="n">checkout</span> <span class="n">tags</span><span class="o">/</span><span class="mf">0.98.1</span>
+</pre></div>
<p>If you have already been building versions of HBase, remove the existing
set of artifacts for safety:</p>
-<pre class="codehilite"><code>rm -rf ~/.m2/repository/org/apache/hbase/</code></pre>
+<div class="codehilite"><pre><span class="n">rm</span> <span class="o">-</span><span class="n">rf</span> <span class="o">~/</span><span class="p">.</span><span class="n">m2</span><span class="o">/</span><span class="n">repository</span><span class="o">/</span><span class="n">org</span><span class="o">/</span><span class="n">apache</span><span class="o">/</span><span class="n">hbase</span><span class="o">/</span>
+</pre></div>
<p>The maven command for building hbase artifacts against this hadoop version is </p>
-<pre class="codehilite"><code>mvn clean install assembly:single -DskipTests -Dmaven.javadoc.skip=true</code></pre>
+<div class="codehilite"><pre><span class="n">mvn</span> <span class="n">clean</span> <span class="n">install</span> <span class="n">assembly</span><span class="o">:</span><span class="n">single</span> <span class="o">-</span><span class="n">DskipTests</span> <span class="o">-</span><span class="n">Dmaven</span><span class="p">.</span><span class="n">javadoc</span><span class="p">.</span><span class="n">skip</span><span class="o">=</span><span class="nb">true</span>
+</pre></div>
<p>To use a different version of Hadoop from that defined in the <code>hadoop-two.version</code>
property of<code>/pom.xml</code>:</p>
-<pre class="codehilite"><code>mvn clean install assembly:single -DskipTests -Dmaven.javadoc.skip=true -Dhadoop-two.version=$HADOOP_VERSION</code></pre>
+<div class="codehilite"><pre><span class="n">mvn</span> <span class="n">clean</span> <span class="n">install</span> <span class="n">assembly</span><span class="o">:</span><span class="n">single</span> <span class="o">-</span><span class="n">DskipTests</span> <span class="o">-</span><span class="n">Dmaven</span><span class="p">.</span><span class="n">javadoc</span><span class="p">.</span><span class="n">skip</span><span class="o">=</span><span class="nb">true</span> <span class="o">-</span><span class="n">Dhadoop</span><span class="o">-</span><span class="n">two</span><span class="p">.</span><span class="n">version</span><span class="o">=</span><span class="err">$</span><span class="n">HADOOP_VERSION</span>
+</pre></div>
<p>This will create an hbase <code>tar.gz</code> file in the directory <code>hbase-assembly/target/</code>
in the hbase source tree. </p>
-<pre class="codehilite"><code>export HBASE_VERSION=0.98.1
+<div class="codehilite"><pre><span class="n">export</span> <span class="n">HBASE_VERSION</span><span class="o">=</span><span class="mf">0.98.1</span>
-pushd hbase-assembly/target
-gunzip hbase-$HBASE_VERSION-bin.tar.gz
-tar -xvf hbase-$HBASE_VERSION-bin.tar
-gzip hbase-$HBASE_VERSION-bin.tar
-popd</code></pre>
+<span class="n">pushd</span> <span class="n">hbase</span><span class="o">-</span><span class="n">assembly</span><span class="o">/</span><span class="n">target</span>
+<span class="n">gunzip</span> <span class="n">hbase</span><span class="o">-</span><span class="err">$</span><span class="n">HBASE_VERSION</span><span class="o">-</span><span class="n">bin</span><span class="p">.</span><span class="n">tar</span><span class="p">.</span><span class="n">gz</span>
+<span class="n">tar</span> <span class="o">-</span><span class="n">xvf</span> <span class="n">hbase</span><span class="o">-</span><span class="err">$</span><span class="n">HBASE_VERSION</span><span class="o">-</span><span class="n">bin</span><span class="p">.</span><span class="n">tar</span>
+<span class="n">gzip</span> <span class="n">hbase</span><span class="o">-</span><span class="err">$</span><span class="n">HBASE_VERSION</span><span class="o">-</span><span class="n">bin</span><span class="p">.</span><span class="n">tar</span>
+<span class="n">popd</span>
+</pre></div>
<p>This will create an untarred directory containing
@@ -292,45 +319,53 @@ and downloading then expanding the file.
<p>For more information (including recommended Maven memory configuration options),
see <a href="http://hbase.apache.org/book/build.html">HBase building</a></p>
<p>For building just the JAR files:</p>
-<pre class="codehilite"><code>mvn clean install -DskipTests -Dhadoop.profile=2.0 -Dhadoop-two.version=$HADOOP_VERSION</code></pre>
+<div class="codehilite"><pre><span class="n">mvn</span> <span class="n">clean</span> <span class="n">install</span> <span class="o">-</span><span class="n">DskipTests</span> <span class="o">-</span><span class="n">Dhadoop</span><span class="p">.</span><span class="n">profile</span><span class="o">=</span><span class="mf">2.0</span> <span class="o">-</span><span class="n">Dhadoop</span><span class="o">-</span><span class="n">two</span><span class="p">.</span><span class="n">version</span><span class="o">=</span><span class="err">$</span><span class="n">HADOOP_VERSION</span>
+</pre></div>
<p><em>Tip:</em> you can force set a version in Maven by having it update all the POMs:</p>
-<pre class="codehilite"><code>mvn versions:set -DnewVersion=0.98.1-SNAPSHOT</code></pre>
+<div class="codehilite"><pre><span class="n">mvn</span> <span class="n">versions</span><span class="o">:</span><span class="n">set</span> <span class="o">-</span><span class="n">DnewVersion</span><span class="o">=</span><span class="mf">0.98.1</span><span class="o">-</span><span class="n">SNAPSHOT</span>
+</pre></div>
<h2 id="building-accumulo">Building Accumulo</h2>
<p>Clone accumulo from apache;</p>
-<pre class="codehilite"><code>git clone http://git-wip-us.apache.org/repos/asf/accumulo.git</code></pre>
+<div class="codehilite"><pre><span class="n">git</span> <span class="n">clone</span> <span class="n">http</span><span class="o">:</span><span class="c1">//git-wip-us.apache.org/repos/asf/accumulo.git</span>
+</pre></div>
<p>Check out branch 1.6.1-SNAPSHOT</p>
-<pre class="codehilite"><code>git checkout 1.6.1-SNAPSHOT</code></pre>
+<div class="codehilite"><pre><span class="n">git</span> <span class="n">checkout</span> <span class="mf">1.6.1</span><span class="o">-</span><span class="n">SNAPSHOT</span>
+</pre></div>
<p>In the accumulo project directory, build it</p>
-<pre class="codehilite"><code>mvn clean install -Passemble -DskipTests -Dmaven.javadoc.skip=true \
- -Dhadoop.profile=2</code></pre>
+<div class="codehilite"><pre><span class="n">mvn</span> <span class="n">clean</span> <span class="n">install</span> <span class="o">-</span><span class="n">Passemble</span> <span class="o">-</span><span class="n">DskipTests</span> <span class="o">-</span><span class="n">Dmaven</span><span class="p">.</span><span class="n">javadoc</span><span class="p">.</span><span class="n">skip</span><span class="o">=</span><span class="nb">true</span> \
+ <span class="o">-</span><span class="n">Dhadoop</span><span class="p">.</span><span class="n">profile</span><span class="o">=</span><span class="mi">2</span>
+</pre></div>
<p>The default Hadoop version for accumulo-1.6.1 is hadoop 2.4.0; to build
against a different version use the command</p>
-<pre class="codehilite"><code>mvn clean install -Passemble -DskipTests -Dmaven.javadoc.skip=true \
- -Dhadoop.profile=2 -Dhadoop.version=$HADOOP_VERSION</code></pre>
+<div class="codehilite"><pre><span class="n">mvn</span> <span class="n">clean</span> <span class="n">install</span> <span class="o">-</span><span class="n">Passemble</span> <span class="o">-</span><span class="n">DskipTests</span> <span class="o">-</span><span class="n">Dmaven</span><span class="p">.</span><span class="n">javadoc</span><span class="p">.</span><span class="n">skip</span><span class="o">=</span><span class="nb">true</span> \
+ <span class="o">-</span><span class="n">Dhadoop</span><span class="p">.</span><span class="n">profile</span><span class="o">=</span><span class="mi">2</span> <span class="o">-</span><span class="n">Dhadoop</span><span class="p">.</span><span class="n">version</span><span class="o">=</span><span class="err">$</span><span class="n">HADOOP_VERSION</span>
+</pre></div>
<p>This creates an accumulo tar.gz file in <code>assemble/target/</code>. Extract this
to create an expanded directory</p>
-<pre class="codehilite"><code>accumulo/assemble/target/accumulo-1.6.1-SNAPSHOT-bin.tar.gz</code></pre>
+<div class="codehilite"><pre><span class="n">accumulo</span><span class="o">/</span><span class="n">assemble</span><span class="o">/</span><span class="n">target</span><span class="o">/</span><span class="n">accumulo</span><span class="o">-</span><span class="mf">1.6.1</span><span class="o">-</span><span class="n">SNAPSHOT</span><span class="o">-</span><span class="n">bin</span><span class="p">.</span><span class="n">tar</span><span class="p">.</span><span class="n">gz</span>
+</pre></div>
<p>This can be done with the command sequence</p>
-<pre class="codehilite"><code>export ACCUMULO_VERSION=1.6.1-SNAPSHOT
+<div class="codehilite"><pre><span class="n">export</span> <span class="n">ACCUMULO_VERSION</span><span class="o">=</span><span class="mf">1.6.1</span><span class="o">-</span><span class="n">SNAPSHOT</span>
-pushd assemble/target/
-gunzip -f accumulo-$ACCUMULO_VERSION-bin.tar.gz
-tar -xvf accumulo-$ACCUMULO_VERSION-bin.tar.gz
-popd</code></pre>
+<span class="n">pushd</span> <span class="n">assemble</span><span class="o">/</span><span class="n">target</span><span class="o">/</span>
+<span class="n">gunzip</span> <span class="o">-</span><span class="n">f</span> <span class="n">accumulo</span><span class="o">-</span><span class="err">$</span><span class="n">ACCUMULO_VERSION</span><span class="o">-</span><span class="n">bin</span><span class="p">.</span><span class="n">tar</span><span class="p">.</span><span class="n">gz</span>
+<span class="n">tar</span> <span class="o">-</span><span class="n">xvf</span> <span class="n">accumulo</span><span class="o">-</span><span class="err">$</span><span class="n">ACCUMULO_VERSION</span><span class="o">-</span><span class="n">bin</span><span class="p">.</span><span class="n">tar</span><span class="p">.</span><span class="n">gz</span>
+<span class="n">popd</span>
+</pre></div>
<p>Note that the final location of the accumulo files is needed for the configuration,
@@ -340,47 +375,48 @@ a path such as <code>target/accumulo-$AC
<h3 id="configuring-slider-to-locate-the-relevant-artifacts">Configuring Slider to locate the relevant artifacts</h3>
<p>You must have the file <code>src/test/resources/slider-test.xml</code> (this
is ignored by git), declaring where HBase, accumulo, Hadoop and zookeeper are:</p>
-<pre class="codehilite"><code><configuration>
+<div class="codehilite"><pre><span class="nt"><configuration></span>
- <property>
- <name>slider.test.hbase.home</name>
- <value>/home/slider/hbase/hbase-assembly/target/hbase-0.98.0-SNAPSHOT</value>
- <description>HBASE Home</description>
- </property>
-
- <property>
- <name>slider.test.hbase.tar</name>
- <value>/home/slider/hbase/hbase-assembly/target/hbase-0.98.0-SNAPSHOT-bin.tar.gz</value>
- <description>HBASE archive URI</description>
- </property>
-
- <property>
- <name>slider.test.accumulo.home</name>
- <value>/home/slider/accumulo/assemble/target/accumulo-1.6.1-SNAPSHOT/</value>
- <description>Accumulo Home</description>
- </property>
-
- <property>
- <name>slider.test.accumulo.tar</name>
- <value>/home/slider/accumulo/assemble/target/accumulo-1.6.1-SNAPSHOT-bin.tar.gz</value>
- <description>Accumulo archive URI</description>
- </property>
-
- <property>
- <name>zk.home</name>
- <value>
- /home/slider/Apps/zookeeper</value>
- <description>Zookeeper home dir on target systems</description>
- </property>
-
- <property>
- <name>hadoop.home</name>
- <value>
- /home/slider/hadoop-common/hadoop-dist/target/hadoop-2.3.0</value>
- <description>Hadoop home dir on target systems</description>
- </property>
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>slider.test.hbase.home<span class="nt"></name></span>
+ <span class="nt"><value></span>/home/slider/hbase/hbase-assembly/target/hbase-0.98.0-SNAPSHOT<span class="nt"></value></span>
+ <span class="nt"><description></span>HBASE Home<span class="nt"></description></span>
+ <span class="nt"></property></span>
+
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>slider.test.hbase.tar<span class="nt"></name></span>
+ <span class="nt"><value></span>/home/slider/hbase/hbase-assembly/target/hbase-0.98.0-SNAPSHOT-bin.tar.gz<span class="nt"></value></span>
+ <span class="nt"><description></span>HBASE archive URI<span class="nt"></description></span>
+ <span class="nt"></property></span>
+
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>slider.test.accumulo.home<span class="nt"></name></span>
+ <span class="nt"><value></span>/home/slider/accumulo/assemble/target/accumulo-1.6.1-SNAPSHOT/<span class="nt"></value></span>
+ <span class="nt"><description></span>Accumulo Home<span class="nt"></description></span>
+ <span class="nt"></property></span>
+
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>slider.test.accumulo.tar<span class="nt"></name></span>
+ <span class="nt"><value></span>/home/slider/accumulo/assemble/target/accumulo-1.6.1-SNAPSHOT-bin.tar.gz<span class="nt"></value></span>
+ <span class="nt"><description></span>Accumulo archive URI<span class="nt"></description></span>
+ <span class="nt"></property></span>
+
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>zk.home<span class="nt"></name></span>
+ <span class="nt"><value></span>
+ /home/slider/Apps/zookeeper<span class="nt"></value></span>
+ <span class="nt"><description></span>Zookeeper home dir on target systems<span class="nt"></description></span>
+ <span class="nt"></property></span>
+
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>hadoop.home<span class="nt"></name></span>
+ <span class="nt"><value></span>
+ /home/slider/hadoop-common/hadoop-dist/target/hadoop-2.3.0<span class="nt"></value></span>
+ <span class="nt"><description></span>Hadoop home dir on target systems<span class="nt"></description></span>
+ <span class="nt"></property></span>
-</configuration></code></pre>
+<span class="nt"></configuration></span>
+</pre></div>
<h2 id="debugging-a-failing-test">Debugging a failing test</h2>
@@ -405,7 +441,8 @@ either stream for logging</p>
</li>
</ol>
<p>Example:</p>
-<pre class="codehilite"><code>target/TestLiveRegionService/TestLiveRegionService-logDir-nm-1_0/application_1376095770244_0001/container_1376095770244_0001_01_000001/out.txt</code></pre>
+<div class="codehilite"><pre><span class="n">target</span><span class="o">/</span><span class="n">TestLiveRegionService</span><span class="o">/</span><span class="n">TestLiveRegionService</span><span class="o">-</span><span class="n">logDir</span><span class="o">-</span><span class="n">nm</span><span class="o">-</span><span class="mi">1</span><span class="n">_0</span><span class="o">/</span><span class="n">application_1376095770244_0001</span><span class="o">/</span><span class="n">container_1376095770244_0001_01_000001</span><span class="o">/</span><span class="n">out</span><span class="p">.</span><span class="n">txt</span>
+</pre></div>
<ol>
@@ -417,12 +454,14 @@ either stream for logging</p>
</ol>
<p>This is all a bit complicated -debugging is simpler if a single test is run at a
time, which is straightforward</p>
-<pre class="codehilite"><code>mvn clean test -Dtest=TestLiveRegionService</code></pre>
+<div class="codehilite"><pre><span class="n">mvn</span> <span class="n">clean</span> <span class="n">test</span> <span class="o">-</span><span class="n">Dtest</span><span class="o">=</span><span class="n">TestLiveRegionService</span>
+</pre></div>
<h3 id="building-the-jar-file">Building the JAR file</h3>
<p>You can create the JAR file and set up its directories with</p>
-<pre class="codehilite"><code> mvn package -DskipTests</code></pre>
+<div class="codehilite"><pre> <span class="n">mvn</span> <span class="n">package</span> <span class="o">-</span><span class="n">DskipTests</span>
+</pre></div>
<h1 id="development-notes">Development Notes</h1>
@@ -469,7 +508,8 @@ are too many region servers running, thi
<p>After every test run: do a <code>jps -v</code> to look for any leftover HBase services
-and kill them.</p>
<p>Here is a handy bash command to do this</p>
-<pre class="codehilite"><code>jps -l | grep HRegion | awk '{print $1}' | xargs kill -9</code></pre>
+<div class="codehilite"><pre><span class="n">jps</span> <span class="o">-</span><span class="n">l</span> <span class="o">|</span> <span class="n">grep</span> <span class="n">HRegion</span> <span class="o">|</span> <span class="n">awk</span> <span class="err">'</span><span class="p">{</span><span class="n">print</span> <span class="err">$</span><span class="mi">1</span><span class="p">}</span><span class="err">'</span> <span class="o">|</span> <span class="n">xargs</span> <span class="n">kill</span> <span class="o">-</span><span class="mi">9</span>
+</pre></div>
<h2 id="groovy">Groovy</h2>
@@ -479,11 +519,12 @@ used Groovy on the production source, th
a Java-only production codebase.</p>
<h2 id="maven-utils">Maven utils</h2>
<p>Here are some handy aliases to make maven easier </p>
-<pre class="codehilite"><code>alias mci='mvn clean install -DskipTests'
-alias mi='mvn install -DskipTests'
-alias mvct='mvn clean test'
-alias mvnsite='mvn site:site -Dmaven.javadoc.skip=true'
-alias mvt='mvn test'</code></pre>
+<div class="codehilite"><pre><span class="n">alias</span> <span class="n">mci</span><span class="o">=</span><span class="err">'</span><span class="n">mvn</span> <span class="n">clean</span> <span class="n">install</span> <span class="o">-</span><span class="n">DskipTests</span><span class="err">'</span>
+<span class="n">alias</span> <span class="n">mi</span><span class="o">=</span><span class="err">'</span><span class="n">mvn</span> <span class="n">install</span> <span class="o">-</span><span class="n">DskipTests</span><span class="err">'</span>
+<span class="n">alias</span> <span class="n">mvct</span><span class="o">=</span><span class="err">'</span><span class="n">mvn</span> <span class="n">clean</span> <span class="n">test</span><span class="err">'</span>
+<span class="n">alias</span> <span class="n">mvnsite</span><span class="o">=</span><span class="err">'</span><span class="n">mvn</span> <span class="n">site</span><span class="o">:</span><span class="n">site</span> <span class="o">-</span><span class="n">Dmaven</span><span class="p">.</span><span class="n">javadoc</span><span class="p">.</span><span class="n">skip</span><span class="o">=</span><span class="nb">true</span><span class="err">'</span>
+<span class="n">alias</span> <span class="n">mvt</span><span class="o">=</span><span class="err">'</span><span class="n">mvn</span> <span class="n">test</span><span class="err">'</span>
+</pre></div>
</div>
<div id="footer">
Modified: incubator/slider/site/content/developing/functional_tests.html
URL: http://svn.apache.org/viewvc/incubator/slider/site/content/developing/functional_tests.html?rev=1608633&r1=1608632&r2=1608633&view=diff
==============================================================================
--- incubator/slider/site/content/developing/functional_tests.html (original)
+++ incubator/slider/site/content/developing/functional_tests.html Tue Jul 8 00:49:34 2014
@@ -171,10 +171,11 @@ Latest release: <strong>0.30-incubating<
-->
<h1 id="testing-apache-slider">Testing Apache Slider</h1>
-<pre class="codehilite"><code> The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL
- NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and
- "OPTIONAL" in this document are to be interpreted as described in
- RFC 2119.</code></pre>
+<div class="codehilite"><pre> <span class="n">The</span> <span class="n">key</span> <span class="n">words</span> <span class="s">"MUST"</span><span class="p">,</span> <span class="s">"MUST NOT"</span><span class="p">,</span> <span class="s">"REQUIRED"</span><span class="p">,</span> <span class="s">"SHALL"</span><span class="p">,</span> <span class="s">"SHALL</span>
+ <span class="n">NOT</span><span class="s">", "</span><span class="n">SHOULD</span><span class="s">", "</span><span class="n">SHOULD</span> <span class="n">NOT</span><span class="s">", "</span><span class="n">RECOMMENDED</span><span class="s">", "</span><span class="n">MAY</span><span class="s">", and</span>
+ <span class="s">"OPTIONAL"</span> <span class="n">in</span> <span class="n">this</span> <span class="n">document</span> <span class="n">are</span> <span class="n">to</span> <span class="n">be</span> <span class="n">interpreted</span> <span class="n">as</span> <span class="n">described</span> <span class="n">in</span>
+ <span class="n">RFC</span> <span class="mf">2119.</span>
+</pre></div>
<h1 id="functional-tests">Functional Tests</h1>
@@ -201,11 +202,13 @@ directory</li>
If it is not present, the tests will fail</p>
<p>The path to the configuration directory must be supplied in the property
<code>slider.conf.dir</code> which can be set on the command line</p>
-<pre class="codehilite"><code>mvn test -Dslider.conf.dir=src/test/configs/sandbox/slider</code></pre>
+<div class="codehilite"><pre><span class="n">mvn</span> <span class="n">test</span> <span class="o">-</span><span class="n">Dslider</span><span class="p">.</span><span class="n">conf</span><span class="p">.</span><span class="n">dir</span><span class="o">=</span><span class="n">src</span><span class="o">/</span><span class="n">test</span><span class="o">/</span><span class="n">configs</span><span class="o">/</span><span class="n">sandbox</span><span class="o">/</span><span class="n">slider</span>
+</pre></div>
<p>It can also be set in the (optional) file <code>slider-funtest/build.properties</code>:</p>
-<pre class="codehilite"><code>slider.conf.dir=src/test/configs/sandbox/slider</code></pre>
+<div class="codehilite"><pre><span class="n">slider</span><span class="p">.</span><span class="n">conf</span><span class="p">.</span><span class="n">dir</span><span class="o">=</span><span class="n">src</span><span class="o">/</span><span class="n">test</span><span class="o">/</span><span class="n">configs</span><span class="o">/</span><span class="n">sandbox</span><span class="o">/</span><span class="n">slider</span>
+</pre></div>
<p>This file is loaded whenever a slider build or test run takes place</p>
@@ -215,15 +218,17 @@ Accumulo tests, as well as a common set
<h2 id="disabling-the-functional-tests-entirely">Disabling the functional tests entirely</h2>
<p>All functional tests which require a live YARN cluster
can be disabled through the property <code>slider.funtest.enabled</code></p>
-<pre class="codehilite"><code><property>
- <name>slider.funtest.enabled</name>
- <value>false</value>
-</property></code></pre>
+<div class="codehilite"><pre><span class="nt"><property></span>
+ <span class="nt"><name></span>slider.funtest.enabled<span class="nt"></name></span>
+ <span class="nt"><value></span>false<span class="nt"></value></span>
+<span class="nt"></property></span>
+</pre></div>
<p>There is a configuration do do exactly this in
<code>src/test/configs/offline/slider</code>:</p>
-<pre class="codehilite"><code>slider.conf.dir=src/test/configs/offline/slider</code></pre>
+<div class="codehilite"><pre><span class="n">slider</span><span class="p">.</span><span class="n">conf</span><span class="p">.</span><span class="n">dir</span><span class="o">=</span><span class="n">src</span><span class="o">/</span><span class="n">test</span><span class="o">/</span><span class="n">configs</span><span class="o">/</span><span class="n">offline</span><span class="o">/</span><span class="n">slider</span>
+</pre></div>
<p>Tests which do not require a live YARN cluster will still run;
@@ -231,36 +236,37 @@ these verify that the <code>bin/slider</
<h3 id="non-mandatory-options">Non-mandatory options</h3>
<p>The following test options may be added to <code>slider-client.xml</code> if the defaults
need to be changed</p>
-<pre class="codehilite"><code><property>
- <name>slider.test.zkhosts</name>
- <description>comma separated list of ZK hosts</description>
- <value>localhost</value>
-</property>
-
-<property>
- <name>slider.test.thaw.wait.seconds</name>
- <description>Time to wait in seconds for a thaw to result in a running AM</description>
- <value>60000</value>
-</property>
-
-<property>
- <name>slider.test.freeze.wait.seconds</name>
- <description>Time to wait in seconds for a freeze to halt the cluster</description>
- <value>60000</value>
-</property>
-
- <property>
- <name>slider.test.timeout.millisec</name>
- <description>Time out in milliseconds before a test is considered to have failed.
- There are some maven properties which also define limits and may need adjusting</description>
- <value>180000</value>
-</property>
-
- <property>
- <name>slider.test.yarn.ram</name>
- <description>Size in MB to ask for containers</description>
- <value>192</value>
-</property></code></pre>
+<div class="codehilite"><pre><span class="nt"><property></span>
+ <span class="nt"><name></span>slider.test.zkhosts<span class="nt"></name></span>
+ <span class="nt"><description></span>comma separated list of ZK hosts<span class="nt"></description></span>
+ <span class="nt"><value></span>localhost<span class="nt"></value></span>
+<span class="nt"></property></span>
+
+<span class="nt"><property></span>
+ <span class="nt"><name></span>slider.test.thaw.wait.seconds<span class="nt"></name></span>
+ <span class="nt"><description></span>Time to wait in seconds for a thaw to result in a running AM<span class="nt"></description></span>
+ <span class="nt"><value></span>60000<span class="nt"></value></span>
+<span class="nt"></property></span>
+
+<span class="nt"><property></span>
+ <span class="nt"><name></span>slider.test.freeze.wait.seconds<span class="nt"></name></span>
+ <span class="nt"><description></span>Time to wait in seconds for a freeze to halt the cluster<span class="nt"></description></span>
+ <span class="nt"><value></span>60000<span class="nt"></value></span>
+<span class="nt"></property></span>
+
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>slider.test.timeout.millisec<span class="nt"></name></span>
+ <span class="nt"><description></span>Time out in milliseconds before a test is considered to have failed.
+ There are some maven properties which also define limits and may need adjusting<span class="nt"></description></span>
+ <span class="nt"><value></span>180000<span class="nt"></value></span>
+<span class="nt"></property></span>
+
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>slider.test.yarn.ram<span class="nt"></name></span>
+ <span class="nt"><description></span>Size in MB to ask for containers<span class="nt"></description></span>
+ <span class="nt"><value></span>192<span class="nt"></value></span>
+<span class="nt"></property></span>
+</pre></div>
<p>Note that while the same properties need to be set in
@@ -284,11 +290,12 @@ away</li>
mvn test -Dslider.conf.dir=../src/test/clusters/remote/slider -Dtest=TestAppsThroughAgent -DfailIfNoTests=false</code></p>
<p><strong>Enable/Execute the tests</strong></p>
<p>To enable the test ensure that <em>slider.test.agent.enabled</em> is set to <em>true</em>.</p>
-<pre class="codehilite"><code><property>
- <name>slider.test.agent.enabled</name>
- <description>Flag to enable/disable Agent tests</description>
- <value>true</value>
-</property></code></pre>
+<div class="codehilite"><pre><span class="nt"><property></span>
+ <span class="nt"><name></span>slider.test.agent.enabled<span class="nt"></name></span>
+ <span class="nt"><description></span>Flag to enable/disable Agent tests<span class="nt"></description></span>
+ <span class="nt"><value></span>true<span class="nt"></value></span>
+<span class="nt"></property></span>
+</pre></div>
<p><strong>Test setup</strong></p>
@@ -313,93 +320,100 @@ mvn test -Dslider.conf.dir=../src/test/c
<li>hdfs dfs -copyFromLocal SLIDER_INSTALL_LOC/agent/conf/agent.ini /slider/agent/conf</li>
</ul>
<p>Ensure correct host name is provided for the agent tarball.</p>
-<pre class="codehilite"><code><property>
- <name>slider.test.agent.tar</name>
- <description>Path to the Agent Tar file in HDFS</description>
- <value>hdfs://NN_HOSTNAME:8020/slider/agent/slider-agent.tar.gz</value>
-</property></code></pre>
+<div class="codehilite"><pre><span class="nt"><property></span>
+ <span class="nt"><name></span>slider.test.agent.tar<span class="nt"></name></span>
+ <span class="nt"><description></span>Path to the Agent Tar file in HDFS<span class="nt"></description></span>
+ <span class="nt"><value></span>hdfs://NN_HOSTNAME:8020/slider/agent/slider-agent.tar.gz<span class="nt"></value></span>
+<span class="nt"></property></span>
+</pre></div>
<h3 id="hbase-tests">HBase Tests</h3>
<p>The HBase tests can be enabled or disabled</p>
-<pre class="codehilite"><code><property>
- <name>slider.test.hbase.enabled</name>
- <description>Flag to enable/disable HBase tests</description>
- <value>true</value>
-</property></code></pre>
+<div class="codehilite"><pre><span class="nt"><property></span>
+ <span class="nt"><name></span>slider.test.hbase.enabled<span class="nt"></name></span>
+ <span class="nt"><description></span>Flag to enable/disable HBase tests<span class="nt"></description></span>
+ <span class="nt"><value></span>true<span class="nt"></value></span>
+<span class="nt"></property></span>
+</pre></div>
<p>Mandatory test parameters must be added to <code>slider-client.xml</code></p>
-<pre class="codehilite"><code><property>
- <name>slider.test.hbase.tar</name>
- <description>Path to the HBase Tar file in HDFS</description>
- <value>hdfs://sandbox:8020/user/slider/hbase.tar.gz</value>
-</property>
-
-<property>
- <name>slider.test.hbase.appconf</name>
- <description>Path to the directory containing the HBase application config</description>
- <value>file://${user.dir}/src/test/configs/sandbox/hbase</value>
-</property></code></pre>
+<div class="codehilite"><pre><span class="nt"><property></span>
+ <span class="nt"><name></span>slider.test.hbase.tar<span class="nt"></name></span>
+ <span class="nt"><description></span>Path to the HBase Tar file in HDFS<span class="nt"></description></span>
+ <span class="nt"><value></span>hdfs://sandbox:8020/user/slider/hbase.tar.gz<span class="nt"></value></span>
+<span class="nt"></property></span>
+
+<span class="nt"><property></span>
+ <span class="nt"><name></span>slider.test.hbase.appconf<span class="nt"></name></span>
+ <span class="nt"><description></span>Path to the directory containing the HBase application config<span class="nt"></description></span>
+ <span class="nt"><value></span>file://<span class="cp">${</span><span class="n">user</span><span class="o">.</span><span class="n">dir</span><span class="cp">}</span>/src/test/configs/sandbox/hbase<span class="nt"></value></span>
+<span class="nt"></property></span>
+</pre></div>
<p>Optional parameters: </p>
-<pre class="codehilite"><code> <property>
- <name>slider.test.hbase.launch.wait.seconds</name>
- <description>Time to wait in seconds for HBase to start</description>
- <value>1800</value>
-</property></code></pre>
+<div class="codehilite"><pre> <span class="nt"><property></span>
+ <span class="nt"><name></span>slider.test.hbase.launch.wait.seconds<span class="nt"></name></span>
+ <span class="nt"><description></span>Time to wait in seconds for HBase to start<span class="nt"></description></span>
+ <span class="nt"><value></span>1800<span class="nt"></value></span>
+<span class="nt"></property></span>
+</pre></div>
<h4 id="accumulo-configuration-options">Accumulo configuration options</h4>
<p>Enable/disable the tests</p>
-<pre class="codehilite"><code> <property>
- <name>slider.test.accumulo.enabled</name>
- <description>Flag to enable/disable Accumulo tests</description>
- <value>true</value>
- </property></code></pre>
+<div class="codehilite"><pre> <span class="nt"><property></span>
+ <span class="nt"><name></span>slider.test.accumulo.enabled<span class="nt"></name></span>
+ <span class="nt"><description></span>Flag to enable/disable Accumulo tests<span class="nt"></description></span>
+ <span class="nt"><value></span>true<span class="nt"></value></span>
+ <span class="nt"></property></span>
+</pre></div>
<p>Optional parameters</p>
-<pre class="codehilite"><code> <property>
- <name>slider.test.accumulo.launch.wait.seconds</name>
- <description>Time to wait in seconds for Accumulo to start</description>
- <value>1800</value>
- </property></code></pre>
+<div class="codehilite"><pre> <span class="nt"><property></span>
+ <span class="nt"><name></span>slider.test.accumulo.launch.wait.seconds<span class="nt"></name></span>
+ <span class="nt"><description></span>Time to wait in seconds for Accumulo to start<span class="nt"></description></span>
+ <span class="nt"><value></span>1800<span class="nt"></value></span>
+ <span class="nt"></property></span>
+</pre></div>
<h3 id="configuring-the-yarn-cluster-for-tests">Configuring the YARN cluster for tests</h3>
<p>Here are the configuration options we use in <code>yarn-site.xml</code> for testing:</p>
<p>These tell YARN to ignore memory requirements in allocating VMs, and
to keep the log files around after an application run. </p>
-<pre class="codehilite"><code> <property>
- <name>yarn.scheduler.minimum-allocation-mb</name>
- <value>1</value>
- </property>
- <property>
- <description>Whether physical memory limits will be enforced for
+<div class="codehilite"><pre> <span class="nt"><property></span>
+ <span class="nt"><name></span>yarn.scheduler.minimum-allocation-mb<span class="nt"></name></span>
+ <span class="nt"><value></span>1<span class="nt"></value></span>
+ <span class="nt"></property></span>
+ <span class="nt"><property></span>
+ <span class="nt"><description></span>Whether physical memory limits will be enforced for
containers.
- </description>
- <name>yarn.nodemanager.pmem-check-enabled</name>
- <value>false</value>
- </property>
- <!-- we really don't want checking here-->
- <property>
- <name>yarn.nodemanager.vmem-check-enabled</name>
- <value>false</value>
- </property>
-
- <!-- how long after a failure to see what is left in the directory-->
- <property>
- <name>yarn.nodemanager.delete.debug-delay-sec</name>
- <value>60000</value>
- </property>
-
- <!--ten seconds before the process gets a -9 -->
- <property>
- <name>yarn.nodemanager.sleep-delay-before-sigkill.ms</name>
- <value>30000</value>
- </property></code></pre>
+ <span class="nt"></description></span>
+ <span class="nt"><name></span>yarn.nodemanager.pmem-check-enabled<span class="nt"></name></span>
+ <span class="nt"><value></span>false<span class="nt"></value></span>
+ <span class="nt"></property></span>
+ <span class="c"><!-- we really don't want checking here--></span>
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>yarn.nodemanager.vmem-check-enabled<span class="nt"></name></span>
+ <span class="nt"><value></span>false<span class="nt"></value></span>
+ <span class="nt"></property></span>
+
+ <span class="c"><!-- how long after a failure to see what is left in the directory--></span>
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>yarn.nodemanager.delete.debug-delay-sec<span class="nt"></name></span>
+ <span class="nt"><value></span>60000<span class="nt"></value></span>
+ <span class="nt"></property></span>
+
+ <span class="c"><!--ten seconds before the process gets a -9 --></span>
+ <span class="nt"><property></span>
+ <span class="nt"><name></span>yarn.nodemanager.sleep-delay-before-sigkill.ms<span class="nt"></name></span>
+ <span class="nt"><value></span>30000<span class="nt"></value></span>
+ <span class="nt"></property></span>
+</pre></div>
<h2 id="testing-against-a-secure-cluster">Testing against a secure cluster</h2>
@@ -412,7 +426,8 @@ the tests has access to HDFS and YARN.</
<p>If there are problems authenticating (including the cluster being offline)
the tests appear to hang</p>
<h3 id="validating-the-configuration">Validating the configuration</h3>
-<pre class="codehilite"><code>mvn test -Dtest=TestBuildSetup</code></pre>
+<div class="codehilite"><pre><span class="n">mvn</span> <span class="n">test</span> <span class="o">-</span><span class="n">Dtest</span><span class="o">=</span><span class="n">TestBuildSetup</span>
+</pre></div>
<h3 id="using-relative-paths-in-test-configurations">Using relative paths in test configurations</h3>
@@ -429,11 +444,12 @@ persistent HDFS cluster.</p>
<p>Use the special property <code>slider.test.conf.dir</code> that is set to the path
of the directory, and which can then be used to create an absolute path
from paths relative to the configuration dir:</p>
-<pre class="codehilite"><code><property>
- <name>slider.test.hbase.appconf</name>
- <description>Path to the directory containing the HBase application config</description>
- <value>file://${slider.test.conf.dir}/../hbase</value>
-</property></code></pre>
+<div class="codehilite"><pre><span class="nt"><property></span>
+ <span class="nt"><name></span>slider.test.hbase.appconf<span class="nt"></name></span>
+ <span class="nt"><description></span>Path to the directory containing the HBase application config<span class="nt"></description></span>
+ <span class="nt"><value></span>file://<span class="cp">${</span><span class="n">slider</span><span class="o">.</span><span class="n">test</span><span class="o">.</span><span class="n">conf</span><span class="o">.</span><span class="n">dir</span><span class="cp">}</span>/../hbase<span class="nt"></value></span>
+<span class="nt"></property></span>
+</pre></div>
</li>
@@ -477,7 +493,8 @@ specific hbase or accumulo categories- a
<ol>
<li>
<p>In the root <code>slider</code> directory, build a complete Slider release</p>
-<pre class="codehilite"><code>mvn install -DskipTests</code></pre>
+<div class="codehilite"><pre><span class="n">mvn</span> <span class="n">install</span> <span class="o">-</span><span class="n">DskipTests</span>
+</pre></div>
<ol>
@@ -486,7 +503,8 @@ specific hbase or accumulo categories- a
</li>
<li>
<p>In the <code>slider-funtest</code> dir, run the tests</p>
-<pre class="codehilite"><code>mvn test</code></pre>
+<div class="codehilite"><pre><span class="n">mvn</span> <span class="n">test</span>
+</pre></div>
</li>
@@ -499,7 +517,8 @@ of a previous build of <code>slider-core
up.</p>
<h4 id="to-propagate-changes-in-slider-core-through-to-the-funtest-classes-for">To propagate changes in slider-core through to the funtest classes for</h4>
<p>testing, you must build/install all the slider packages from the root assembly.</p>
-<pre class="codehilite"><code>mvn clean install -DskipTests</code></pre>
+<div class="codehilite"><pre><span class="n">mvn</span> <span class="n">clean</span> <span class="n">install</span> <span class="o">-</span><span class="n">DskipTests</span>
+</pre></div>
<h2 id="limitations-of-slider-funtest">Limitations of slider-funtest</h2>